Sunday, December 13th, 2015
If you click on the Crawl tab on the side bar of GWT, you’ll see a link for robots.txt Tester.
Robots.txt Tester page displays whether you have errors or warnings in your robots.txt file. Sometimes a webmaster may add something to the robots.txt that may disallow Googlebot from accessing your site so your entire website may get de-indexed. It is good to check this page every once in a while.
Filed under: Documentation→Website Performance
Tags: google webmaster tools, gwt, robots.txt, robots.txt test, website performance
1,500,000 Points for
$110 $68. If you're running low on points — don't miss out