Reply
robots.txt
Old 09-30-2006, 11:50 AM robots.txt
snipergrunge's Avatar
Experienced Talker

Posts: 44
I am absolutely unfamiliar with /robots.txt what is it? what should i put and how it is important for search engines?
snipergrunge is offline
Reply With Quote
View Public Profile
 
When You Register, These Ads Go Away!
Old 09-30-2006, 12:00 PM Re: robots.txt
LadynRed's Avatar
Super Moderator

Posts: 6,692
Location: Tennessee
It's not all that important to search engines, they will spider your site w/o one, though if it's there they will use it - or should. There is a ton of information on the internet about robots.txt, do a search for it. The file should go into the root directory of your site. It can be used to STOP spam bots and spiders from indexing all or part of your site. Unfortunately, there are many bots that will ignore the robots.txt file too.

Here are a few sources:
http://www.javascriptkit.com/howto/robots.shtml
http://www.robotcop.org/
http://www.robotstxt.org/wc/robots.html
http://www.robotstxt.org/wc/active/html/index.html
http://www.psychedelix.com/agents/index.shtml
__________________
Web Goddess & Web Standards Evangelist :) - Tables Be Gone !!
"Using or working with IE is like having to wear a 1970's polyester suit with pantyhose and a girdle, to work everyday"
Carolina Corvette Club
LadynRed is offline
Reply With Quote
View Public Profile
 
Old 09-30-2006, 01:31 PM Re: robots.txt
stOx's Avatar
Machine

Latest Blog Post:
Worlds Smallest Car - Peel P50
Posts: 2,114
Name: Matt. (>',')>
Location: London, England.
You only need one if you want to stop bots from accessing certain parts of your site.
stOx is offline
Reply With Quote
View Public Profile Visit stOx's homepage!
 
Old 10-02-2006, 01:45 AM Re: robots.txt
Novice Talker

Posts: 6
www.robotstxt.org
nizhongmin is offline
Reply With Quote
View Public Profile
 
Old 10-02-2006, 05:27 AM Re: robots.txt
Kirtan's Avatar
Who Am I?

Posts: 377
Name: Venkat Raj
Location: Salem, South India
Generally speaking, whenever a spider visits your page, it follows all the links in your page. so all of your pages linked in a logical manner then all the pages in your site will be indexed by that particular search engines. If you want to allow spiders to index particular pages and not to index some other area of your site then you have to use robots.txt.
For example, you may not want to allow spiders to index your images, scripts,etc for that you have to disallow spider using robots.tx
__________________
All the Buddhas of all the ages have been telling you a very simple fact: Be -- don't try to become.
Kirtan is offline
Reply With Quote
View Public Profile Visit Kirtan's homepage!
 
Reply     « Reply to robots.txt
 

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off




   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML

 


Page generated in 0.29881 seconds with 12 queries