We all know about googlebot (Google), msnbot (MSN), Slurp (Yahoo) and Teoma (Ask). All of these benevolent robots, listens to us and respects our wishes about parts of our website that we don’t want to be crawled. We can just give all of them just a robots.txt file explaining what we want, and he’ll happily comply. But what if you’re intimidated by the idea of communicating directly with these robots? After all, not all of us are fluent and well known in the language of robots.txt.

This is why Google introduced personal robot translator in google webmaster tools. This will help you to make your robots.txt file even if you don’t know anything about it. It’s designed to give you an easy and interactive way to build a robots.txt file. It can be as simple as entering the files and directories you don’t want crawled by any robots.

Once you’re finished with the generator, you can test the effects of your new robots.txt file with our robots.txt analysis tool. When you’re done, just save the generated file to the top level (root) directory of your site, and you’re set to go.

There are a couple of important things to keep in mind about robots.txt files:

• Not every search engine will support every extension to robots.txt files

The Robots.txt Generator creates files that Googlebot will understand, and most other major robots will understand them too. But it’s possible that some robots won’t understand all of the robots.txt features that the generator uses.

• Robots.txt is simply a request

Although it’s highly unlikely from a major search engine, there are some unscrupulous robots that may ignore the contents of robots.txt and crawl blocked areas anyway. If you have sensitive content that you need to protect completely, you should put it behind password protection rather than relying on robots.txt.
We hope this new tool helps you communicate your wishes to Googlebot and other robots that visit your site.

Popularity: 80% [?]