#1 (permalink)

I probably already know the answer to this question but what the hell...

Can I do this in my robots.txt file ?

User-agent: robot-I-want-to-crawl-my-site.
allow: /

User-agent: *
Disallow: /


The objective is to give permission to the robots I like.
And disallow all the rest.

#2 (permalink)

Nope

robots.txt is an exclusion protocol only. It is also a voluntary protocol to follow so not all bots honour the exclusions.

__________________
Chris. ->> Links are advertising NOT optimising!! <<-
Indifference will be the downfall of mankind, but who cares?
Code Samples | People Counting System

#3 (permalink)

I had hope that I was wrong.
C'est la vie.

OK,
In this case anyone have a master list of american search engines ?
I'm going to have to do this the hard way.

#4 (permalink)

What is the benefit of Robots.txt file?

#5 (permalink)

[quote]
What is the benefit of Robots.txt file?
[/quotes]

You can use it to give hints on the search engines agents, and blacklist some parts of your site, for example.
You can see the robots.txt of this site there:
http://www.webmaster-talk.com/robots.txt

__________________
Listen to the ducky: "This is awesome!!!"

#6 (permalink)

No. But can you use geo-IP blocking instead? You can actually ban robots (return an unauthorized message) as opposed to hanging a "do not enter" sign.

__________________
4 ways to improve the lives of the "bottom billion"

"HEY YOU KIDS GET OFF MY LAWN!" -John McCain

#7 (permalink)

Although Google, Yahoo, Ask etc. have some extensions.

Here's my blog post about having the robots.txt link xml sitemap file.

Basicly you can now write: Sitemap: http://www.example.com/sitemap.xml

__________________
Promote Websites | Keyword Selection | HTML Sitemap | XML Sitemaps Generator

#8 (permalink)

Why would you want to disclude something from showing from your website. please let me know as I would like to know the functions of this protyocol.

#9 (permalink)

There are many reason you would want to exclude pages or folders from a site.

eg:
Many forums have several ways of getting to the same page, using the disallow you can stop compliant bots from accessing the print versions

If you have tracking links attached to many of your external links you can exclude these with;
disallow: /folder/pagename.ext?track=*

see http://www.robotstxt.org for more on the protocol and http://www.highrankings.com/forum/in...p?showforum=62 for more examples and instances.

__________________
Chris. ->> Links are advertising NOT optimising!! <<-
Indifference will be the downfall of mankind, but who cares?
Code Samples | People Counting System

#10 (permalink)

Ditto. Can't include permission on robots.txt - they open themselves to anything and everything on your server if you don't exclude it!

__________________
Green talkupation is always appreciated. =]
Free Online Games - Cheap Power Leveling - Pontiac Grand Am