Posts: 377
Name: Venkat Raj
Location: Salem, South India
|
Generally speaking, whenever a spider visits your page, it follows all the links in your page. so all of your pages linked in a logical manner then all the pages in your site will be indexed by that particular search engines. If you want to allow spiders to index particular pages and not to index some other area of your site then you have to use robots.txt.
For example, you may not want to allow spiders to index your images, scripts,etc for that you have to disallow spider using robots.tx
__________________
All the Buddhas of all the ages have been telling you a very simple fact: Be -- don't try to become.
|