Florida Search Engine Optimization

XML Sitemap Generator for WordPress

I recently decided to install the XML site map generat0r by Arne Brachold. The installation went fairly well, however, I have received error when attempting to verify the site map in Google Web Master Tools. Although the error was shown to be from a exclusion in the Robots.txt file, there should not have be an error. The robots.txt file was revamped and tested in the Google yet their were numerous attempts before I realized that in this instance, I had to specifically create a rule for Googlebot in the robots.txt file…

useragent: Googlebot
Disallow:

I find this rather strange. I typically write something along the lines of …

useragent: *
Disallow:/foo/
Disallow:/bar/

I haven’t had any problems with sitemaps using the previous method and it wasn’t until finding the article on Webmaster Help Center
“- How do I block Googlebot? “, www.google.com/support/webmasters/bin/answer.py?answer=40364

“Googlebot follows the line directed at it, rather than the line directed at everyone.”

Food for thought and spiders.

0 comments

There are no comments yet...

Kick things off by filling out the form below.

Leave a Comment