robots.txt is needed if you would like to stop search engines indexing a certain part of your site, assuming that they conform to the rules of course..
That thing is pretty cool, showed me some things I didn't know that spiders could see about my sites, and also showed me things I thought they were seeing and not :-P
That thing is pretty cool, showed me some things I didn't know that spiders could see about my sites, and also showed me things I thought they were seeing and not :-P
well, it seems to me that the tool is a bit to strict.. it isn't showing navigation of sites that I know for sure get indexed perfectly....