Getting more information from a Visiting BOT

**bbutton** · 01-09-2007, 05:04 AM

Hi,

Does anyone know if you can find out from a visiting bot that has a 404 which url had the broken link on it.

I have used
request.servervariables("ALL_HTTP") on my 404 page but this does not contain the referring URL that the BOT found the broken link on.

Kind regards
Bill Button
Technical Manger
http://www.rodmatic.com

**ForrestCroce** · 01-13-2007, 07:15 PM

Quote:

Originally Posted by bbutton

Hi,

Does anyone know if you can find out from a visiting bot that has a 404 which url had the broken link on it.

I have used
request.servervariables("ALL_HTTP") on my 404 page but this does not contain the referring URL that the BOT found the broken link on.

Kind regards
Bill Button
Technical Manger
http://www.rodmatic.com

Doubtful; this should be in the referrer field of the request header, but if a bot doesn't want to talk to you, it won't.

But you should have the URL that came up 404 in your web stats. I'd do a search on that ( either through Google, or if it's an intranet site, using Windows Search ) and see if you can find out. I'd try the full URL and also just the name of the file the specific page lives in, in case it's a relative link.

**chrishirst** · 01-25-2007, 05:26 PM

Extract it from the 404 page

the originating error page is found in the querystring to the error page NOT the HTTP response headers. Use request.querystring to retrieve it

**bbutton** · 01-26-2007, 11:40 AM

Hi

I already gather this info, but it is blank when googlebot comes visiting, what I am hoping to find but I do not think it is possible. Is to find the page that GoogleBot found the link from.

**chrishirst** · 01-26-2007, 11:55 AM

bots don't "click" links to find your site so don't have a referrer. It's simply following an entry in the database. If the URL is incorrect make sure that it returns a 404 (most custom 404 pages don't) and it will eventually stop calling that URL.

Alternatively use a 301 redirect to send it to the correct URL

**ForrestCroce** · 01-26-2007, 11:56 PM

Actually, even though one googlebot pulled the link out of a database, another one found it somewhere while populating that database. But you're right that it doesn't just click through, and it's probably not possible ( outside Google's data banks ) to find out where it was discovered.