A robots.txt file restricts access to your site by search engine spiders that crawl the web. The spiders, before they access pages of a site, check to see if a robots.txt file exists that prevents them from accessing certain pages.
You need a robots.txt file only if your site includes content that you don’t want search engines to index. How can you check that the robots.txt file has been correctly set up ?
Fortunately, there is a very useful tool which allows you to do it in an easy way.
Just open http://tool.motoricerca.info/robots-checker.phtml
Enter URL like this – http://www.domain.com/robots.txt and click on “check robots.txt” button.
The contents of robots.txt file will be displayed.
For more information visit our website: http://www.seowebplanet.com
