Spiders questions

**execute** · 11-12-2005, 08:55 AM

I have made my own script to log all search engine crawlers in my database each time they come to my website, i get info about which page they go to. But i've noticed one thing: They seem to never contact server, in other words, they never click on links that were automatically generated by a script like PHP.
MSN bot came to my website, and indexed most pages, pretty much the ones in my nav and footer. However Yahoo! Slurp came and they only indexed a folder called / ... i guess the root. I'm listed on yahoo but it didnt index all my pages like msn bot. Google bot never came.

I was also wondering, what other things can i log about Search engine Crawlers that might help me improve my optimization.. My stupid host has Session ID activated for every page, perhaps thats what made yahoo slurp not index my pages?

**execute** · 11-12-2005, 04:44 PM

So i guess they dont click on links that deal with DATABASE? So spiders never actually let the server execute the php or something?

I mean i have links to tutorials in one of my pages i assumed they would index the tutorials too. But it looks like msn bot never visited them. And they are generated by a database. So it doesnt display Database generated links?

**kline11** · 11-12-2005, 08:00 PM

It sounds like you have a lot of dynamic pages. Bots will index them, but slowly. They don't want to get trapped. Be patient, it takes time before a bot will "trust your site" for a deep crawl, especially Google.

**zemez_man** · 11-13-2005, 04:38 AM

oohhh this is a nice information for me ..

**madkad** · 11-14-2005, 07:00 AM

i have php links etc etc they have been indexed it does happen like kline11 says it takes time

one thing to think about small urls do index better and urls that dont contain(?,=,&,_,-,+) will index even better some people have started to make scripts were there urls are less php format and more HTML format which index realy quick and well but isnt a easy job and can be time consuming

well hope that helps

**execute** · 11-14-2005, 04:42 PM

yea i already made all my links XHTML compatible, so any links generated by PHP are using entities for &.
I guess i need to turn off this Trans_id thing, but i dont have access to my hosts PHP.ini so im prolly gonna attempt a ini_set