Reply
**** Slurp spiders
Old 06-27-2007, 02:59 AM **** Slurp spiders
Super Talker

Posts: 122
Name: Nate
Location: Orlando, FL
I'm running a vBulletin forum, and i know it happens on other vBulletin forums as well..but there are always dozens and dozens of Yahoo! Slurp Spiders......WHY?!?!

Are they sucking up bandwidth?
NateL is offline
Reply With Quote
View Public Profile
 
When You Register, These Ads Go Away!
     
Old 06-27-2007, 05:20 AM Re: **** Slurp spiders
Extreme Talker

Posts: 243
Location: Belgium, Antwerp, Zoersel
That's because Yahoo is indexing your pages, so people searching for them with Yahoo can actually find them. Of course, it "sucks" some of your bandwidth, but generally not that much.

If you really get a lot of hits from it however, it could be possible that you confused the bot by using a lot of URLs that look the same, but are not. For example, if you use the get method to maintain your sessions, your index page, though it is the same could look like this:

index.php?PHPSESSID=4a54c1f54e4e57b98ae4ea
index.php?PHPSESSID=e7a84f46e8a8e6a4f68ea4
...

To prevent this, make sure you only use cookies to manage you sessions, or to disable sessions completely when a search engine is visiting you. If you really don't want yahoo to index your site, you should be able to disable it in robots.txt.
Orodreth is offline
Reply With Quote
View Public Profile Visit Orodreth's homepage!
 
Old 06-27-2007, 11:59 AM Re: **** Slurp spiders
Banned

Posts: 510
Name: CHRIS
Location: I live in Google's Home State
I would not be really worried about it....I mean it can only last so long, and really If spiders are "sucking your bandwith" you will need a dedicated server soon anyways.
Vasity is offline
Reply With Quote
View Public Profile Visit Vasity's homepage!
 
Old 06-30-2007, 02:42 AM Re: **** Slurp spiders
Novice Talker

Posts: 8
From what I understand, adding a sitemap component such as VBSEO should help save on bandwidth. Of course I've only read this and can't comment on whether or not it's true but it does make sense.
__________________
Family Friendly Support Forum
d00k is offline
Reply With Quote
View Public Profile Visit d00k's homepage!
 
Reply     « Reply to **** Slurp spiders
 

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off




   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML

 


Page generated in 0.11410 seconds with 13 queries