Home > Search Engines > Robots.txt files

Robots.txt Files

Uploaded Robots.txt is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable.

A good resource to help you create a Robots.txt file can be viewed here: http://www.mcanerin.com/en/search-engine/robots-txt.asp

See also