GoogleBot took all my bandwidth

**camperjohn** · 08-23-2005, 05:29 PM

Short version:

- I have 46 websites.

- I was sending out about 10 gigs a month in bandwidth from users surfing.

- In the last 20 days, GoogleBot has crawled 100 gigs of bandwidth off my servers, essentially shutting them down. Many pages twice. I now have a nice "Service suspended" message on all my sites because GoogleBot just kept on going and going and going...

- Therefore 10% of my bandwidth is from real users, 90% is wasted on GoogleBot crawling it.

Obviously I don't want to stop Google from crawling my site, but I need to slow it down somehow when my servers are reactivated or it will happen again.

Suggestions?

JM

**chrishirst** · 08-23-2005, 06:14 PM

Get more bandwith

**kline11** · 08-23-2005, 07:17 PM

HA HA HA.
Sorry I just read the post then Chris' response. This made my day. It reminded me of that old joke where the guy goes to the doctor and says, "Doc you've got to help me!" Doctor: "What is the problem?" Patient: "It hurts when I bend my arm" Doctor: "Then don't bend it".

**DarkLink** · 08-23-2005, 08:40 PM

He did not say to not use googlebot, but I am going to, so do not use googlebot

**monkey** · 08-23-2005, 10:37 PM

have you ever heard of a robot.txt? tell googlebot to only revisit certain pages after certain days or tell it to not visit those pages at all. cut down the amount of pages it wants to crawl, and i imagine that will reduce the bandwith it takes.

**shabbirbhimani** · 08-24-2005, 05:31 AM

Quote:

have you ever heard of a robot.txt?

its robots.txt

Also you can use Meta t o restrict them for say 2-5 mins.

**Skorch** · 08-24-2005, 05:33 AM

**chrishirst** · 08-24-2005, 04:18 PM

revist-after is a meta tag that NO internet bot observes so don't bother with that

You either block Googlebot and save your bandwidth or allow it to crawl. Your choice.

**bulldog1256** · 08-27-2005, 01:45 AM

same problem with the bot. I emailed google and they slowed it down, still visits everyday but doesn't use as much bandwidth. try emailing them

**Mightybb** · 08-27-2005, 04:02 PM

Ouch! Google seems to be alright with me.

**extermin8tor** · 08-29-2005, 05:33 AM

how can google waste that much bandwidth. unless your sites are HUGE. also they dont go on that often do they. or do they.

you could probably edit your robots.txt file to stop them or something.

or just get unlimited banndwidth, it only costs maybe an extra 5 bucks or so a month ive seen

**naplesvacation** · 08-29-2005, 10:42 PM

For problems like this.

Googlebot is your friend - dont make him feel unwanted.

**luke456** · 08-30-2005, 06:36 AM

Quote:

Originally Posted by kline11

"Doc you've got to help me!" Doctor: "What is the problem?" Patient: "It hurts when I bend my arm" Doctor: "Then don't bend it".

sorry, I just have to laugh. LOL