XML Sitemap Generator for WordPress
I recently decided to install the XML site map generat0r by Arne Brachold. The installation went fairly well, however, I have received error when attempting to verify the site map in Google Web Master Tools. Although the error was shown to be from a exclusion in the Robots.txt file, there should not have be an error. The robots.txt file was revamped and tested in the Google yet their were numerous attempts before I realized that in this instance, I had to specifically create a rule for Googlebot in the robots.txt file…
useragent: Googlebot
Disallow:
I find this rather strange. I typically write something along the lines of …
useragent: *
Disallow:/foo/
Disallow:/bar/
I haven’t had any problems with sitemaps using the previous method and it wasn’t until finding the article on Webmaster Help Center
“- How do I block Googlebot? “, www.google.com/support/webmasters/bin/answer.py?answer=40364
“Googlebot follows the line directed at it, rather than the line directed at everyone.”
Food for thought and spiders.

0 comments
Kick things off by filling out the form below.
Leave a Comment