Google PageRank Re-Crawls

**DaveTodd** · 01-04-2003, 08:35 AM

For a while now I've been picking up info. about the Google PageRank feature.

One thing that I've learnt is that pages with different PR scores get recrawled more of less frequently dependant upon their scores.

Some get crawled in the main index, once a month. Those with higher scores get recrawled something like once a week, then twice a week, and every day, as their ranks get higher.

Does anybody know what the PageRank boundaries are for these levels?

eg.

PR > 3 - Only Main Crawl
3 < PR > 4 - Once A Week

etc. etc.

This would be very helpful to me, and I'm sure a lot of other people, I've tried to find info. on the net, but drew a blank.

Many thanks

**Lars** · 01-04-2003, 03:56 PM

I don't know but I just saw that my page went from PR3 to PR4.

**Chopper** · 01-04-2003, 06:04 PM

Im currently on a PR 6 and get craweled every day. When I was on a 7 it was once a day also.

**DaveTodd** · 06:37 PM

Thankx Chopper...

From this we can tell that PR 6+ definately gets crawled every day. Can anyone with PR 5 comment on how frequently they get crawled? The same? Or less?

Chopper, you get crawled every day...does your recrawl get listed in the index every day? I'm guessing so...

Does anyone else have any more info. on this subject? The more we can find out about Google and the PageRank algorithm, the better off we all will be.

More light on this subject needed...

**EDIT**
I was visited by the illustrious GoogleBot on 02/01/03, it requested robots.txt, got and error doc, and left. Does this indicate that robots.txt is a vital file? And that having one can seriously improve your chances of being crawled? Do I need to add any code to my index page / into a robots.txt file to help the Bots to crawl deeper into my site?

With PR 4, 02/01/03 is the only entry in my logs of a visit from GoogleBot thus far this month.

This would show that PR < 4 means a crawl of possibly once a week or less.

So we not know that PR6+ = Daily
PR4- = Weekly or less

Can anyone else with any more info shed any more light on the subject?

**Chopper** · 01-04-2003, 09:08 PM

Quote:

Chopper, you get crawled every day...does your recrawl get listed in the index every day? I'm guessing so...

Yes thats spot on

**Criper2000** · 01-05-2003, 12:55 AM

Well right now im a 4, so cant help you yet

**DaveTodd** · 01-05-2003, 06:59 AM

Having just checked my logs from yesterday, I found the elusive Googlebot had been back.

Once again, it arrived, requested robots.txt, got a 404 error, and left.

I have now uploaded a simple robots.txt file, which can be found at this address to tell all spiders to index all the site.

I'll let you know if this improves anything...

But, it does confirm one thing :

PR 4- = Once a week
PR 6+ = Once a day

We could do with knowing about PR5 if anyone has a page with this rank, and also PR3, so that we can put together the rest of the PageRank mystery...

**david** · 09:00 AM

Hmm...... This is interesting:

The following is selected data from my statistics database. It shows all the times the Googlebot has visited the site this year (I'll keep you updated on this as time passes, though). Both sites are PR6.

The number at the end of the row is the number of pages spidered that day.

www.free-webhosting.info - 2003-01-01 00:06:15 - 144
www.free-webhosting.info - 2003-01-02 01:01:15 - 77
www.freewebmasterhelp.com - 2003-01-03 16:29:50 - 141
www.freewebmasterhelp.com - 2003-01-04 00:00:32 - 464
www.freewebmasterhelp.com - 2003-01-05 00:56:43 - 19
freewebmasterhelp.com - 2003-01-06 23:59:55 - 180
freewebmasterhelp.com - 2003-01-07 08:24:10 - 4

There seem to be some strange patterns here.

Also I've looked at my PR5 site. It looks like it could have had 7 visits by the Googlebot last month, but I don't have complete data for then (only a graph), so I'll do another DB query later in the month and see if I can give any more details.