Hi
What is Robots.txt, I get a message on google robots.txt timeout.

This is probably because I don't have one in the root dir, the sitemap is there so do I have to create a blank text file named robots.txt and place it also in the root dir.


dave

robots.txt is to give instructions to spider bots.
Its mostly used to tell them of any places they arent allowed to go.

When a spider goes to your site it will first look for it to see if a robots.txt is there.
If you dont want to see all those 404 errors it would do no harm to put a blank robots.txt file in your root dir.

Also if you need some help on making w=one this is a good website

http://www.searchengineworld.com/rob...s_tutorial.htm

tell you what to do and why to do it and all you have to do is make it in a normle text file

You can indeed create an empty file. It will at the very least get rid of al the 404 errors in your web logs. I would also suggest making a favicon.ico image for your site and placing it in the root html directory also.

Thanks for the help, I had a look at the webpage suggested, couple of questions - sorry to be a pain.

When I make a txt file - can I use Notepad and do I save it as encoding ANSI, or do I have to use some other format.

I take it that the text below will ban search engines from TestDir folder and also the pages named in the html folder.

# start of robots.txt file

User-agent: *
Disallow: /testdir/
Disallow: /html/test_1.htm
Disallow: /html/test_2.htm

# EOF

When I FTP to the site should I also use ASCII format to save the content.

One other question why should I need to place a favicon.ico in the root dir, and what is it.

Dave

sorry i cant help you on the ASCII thing

but yes you robot txt looks ok to me

and for the favicon.ico

a favicon.ico is that little img you see in the left hand side were the url is on you browser not all sites have them i have one on my site i havent stoped robot going to it and its not give me any problems that i know of so i wouldnt worry about it

You can also generate a robots.txt file here for free:
http://www.searchbliss.com/webmaster...-generator.htm
which includes all of the major bots.

Thanks for the help guys, I uploaded the file and used ASCII format with the FTP not binary, I downloaded the file again to check it had not been corrupted and it was fine..

I'll not bother with the favicon just yet.

Thanks

Dave