Ok i am stumped. I normally enjoy great results on google for my own personal site. Its not a major problem as imy own site is only a side line and not a business site etc & only ever checked by family etc. I do however work part time as a web developer for a small IT company & normally we get great results but I am worried something new is going on & want to make sure it doesnt happen to our customer sites.

I guess I am just curious why a site should all of a sudden drop off the radar & what do I do to ensure it doesnt happen? Especially where the sites that I look after are concerned I would like to know a bit more.

Whats this about a robots.txt file & do you really need one if you arent excluding any pages at all ?

p.s my search is normally for 'tonya' & my site www.tonya.me.uk normally comes in the top 10. For a month it dropped off completely & has now appeared again but way down at 60 something.

Hi Tonya,

You need to edit your robots.txt. Checkout http://www.robotstxt.org/wc/robots.html for further details.

Hi Tonya,

As quoted from Jiten's link:

"You can read the whole standard specification but the basic concept is simple: by writing a structured text file you can indicate to robots that certain parts of your server are off-limits to some or all robots."

Doesn't this mean that if you aren't excluding, you don't need this file?

Gringo - my thoughts exactly. I am not excluding so why do I need it ?
Ive read the bumf on the net on various sites including the one above & get all the info about exclusions etc etc
But still cant see why you need it if there are no exclusions because I cant find anywhere where it says the crawlers or bots from search engines like google still work better with one - despite what google say

you don't need a robots.txt unless you need to exclude something. The advantage of having a blank one is that it prevents your error logs filling up.

No bots do not work better with or without.