|
Dear all,
I would like to confirm with all experts here. Whether Googlebot would crawl all files within a directory if such directory contains file(s) which has/have been linked elsewhere?
It seems to have no good reason for a bot to do so. However, when I checked with my server log, it seems that it is the case. For example,
(a) I put a file under /archive/ and named it as abc.html
(b) I then created a link for the abc.html from my site's front page
(c) what I discovered is that Googlebot searches both /archive/ and /archive/abc.html in its crawl
(d) of course, since I do not have index.html/php under /archive, the server log generated 404 errors.
FYI, my site is PR6 and this does not occur in site with PR5 or lower.
Any one can confirm this?
Regards,
leunga
|
