What is “robots.txt Test” in GWT?

If you click on the Crawl tab on the side bar of GWT, you’ll see a link for robots.txt Tester.
image063

Robots.txt Tester page displays whether you have errors or warnings in your robots.txt file. Sometimes a webmaster may add something to the robots.txt that may disallow Googlebot from accessing your site so your entire website may get de-indexed. It is good to check this page every once in a while.

Filed under: DocumentationWebsite Performance
Tags: , , , ,