Hi

Heard that its possible to easily make a website 'robot friendly' re search engines etc by simply adding in a piece of text

Is this true & if so does anyone know the text to include

Thank you

Hi AJ.B,

most text that you put in your web page will be readable by robots - it's just certain things you need to watch out for like javascript. Robots can't read javascript (at the moment).

Heard that its possible to easily make a website 'robot friendly' re search engines etc by simply adding in a piece of text

absolute rubbish!!

I think you might mean a "robots.txt" file. This file goes in the root of your site and allows you to instruct robots NOT to index certain files or directory contents.

This is the sort of thing you can put in this text file :

User-agent: *
Disallow: somefile.htm
Disallow: /admin/
Disallow: /scripts/
Disallow: /styles/
Disallow: /images/

The above would tell robots NOT to index "somefile.htm" and the contents of the specified directories.
You can give instructions to individual robots like so:

User-agent: scooter

Don't forget the <meta name="robots" ........ line too.

I wouldn't say an exclusion standard is "robot friendly" but I think this is what you have heard about.

Making a site robot friendly is often termed 'Search Engine Optimisation' and WT has a forum all about it right here: http://www.webmaster-talk.com/forumdisplay.php?f=102

Are you talking about rewriting the urls to make them more spider friendly? If so, you need to learn more, or get help on mod_rewrite by visiting that link. It does help somewhat in getting your site indexed more thoroughly.

I have heard, and seen some evidence on my own sites that dynamic urls

www.yoursite.com/script.php?12345&take=54678

are crawled, but not as deep as static urls

www.yoursite.com/script/12345/54678.html

mod_rewrite can do it for you.

Frank