robotstxt.org


Website: http://www.robotstxt.org/


Description: information about robots.txt, the web robots exclusion standard, and writing well-behaved web robots.

Keywords: robots.txt, robots exclusion standard