Found this cool script that shows you the crawler, time, IP address, page requested etc. for each hit.

http://www.darrinward.com/googletrax/

you can also just grep your log file.

from a unix prompt:

cat access_log | grep googlebot | less

yeah thats what i do

cat access_log | grep googlebot | less

It's much easier to use Happy Log or LogAnalyzer.. They know all the spiders.

i should write an app that will parse log files and throw results in a db, so you can check which files were accessed by which spiders, and when.

Also, which visitors came from which search engines.

The only problem I see with this is that everyone's log format is different.

wow great contrib, thanks.

I get a 404 visyting your link Pol