How Set Robots.txt Properly?

**jacobsz** · 06-27-2009, 10:10 AM

can anyone tell me how to set the robots.txt on server?so the google can index all the content of my website?please help..thx before

**jacobsz** · 06-27-2009, 10:29 AM

if i want to block the google from indexing my expired webpage with the robots.txt...what is the command should i write in the robots.txt file,for example if my expired webpage is >www.myweb.com/1_10_Lenovo-ideapad-S10.html. so what should i type in the robots file?

**Lashtal** · 06-28-2009, 02:44 PM

Allow and Disallow....

1) http://sitemaps.blogspot.com/2006/02...stxt-file.html

2) http://www.google.com/support/webmas...n&answer=35302

**Richard_zhou** · 07-16-2009, 05:51 AM

robotstxt.org maybe help u.

**Gorlim** · 07-21-2009, 07:04 AM

I'm afraid if you decide to block expired pages with robots.txt, soon the file will grow too large. The best what you can do is: Move expired pages to /myweb.com/expired/lenovo-bla-bla... and just block the /expired/ folder from indexing.

**chrishirst** · 07-22-2009, 10:02 AM

Quote:

Originally Posted by jacobsz

can anyone tell me how to set the robots.txt on server?so the google can index all the content of my website?please help..thx before

Robots.txt does NOT tell SEs to index your site. It is an exclusion protocol to tell compliant bots what NOT to indexcrawl.
See www.robotstxt.org