New robots.txt tool They use the mini on their website and on their intranet. On their intranet they did not have an robots.txt and th mini did not crawl it until I put an robots.txt that said allow.
Robotstxt.org Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.