Hello,
I have my main domain at public_html/ then I have other subfolders, some of them are a part of the domain and some of them are not, but in future those subfolders will be the root of new domains.

For example public_html/forums is a good folder, part of the www.mydomain.com

But the folder public_html/mynewdomain is not yet a domain, I need access to it by accessing the www.mydomain.com/mynewdomain url, but only temporarly, since I don't have yet a domain for that subfolder.

I would like search engines to NOT access at all to the subfolder public_html/mynewdomain, because I do not want this new site be in google as www.mydomain.com/mynewdomain, instead it should be www.mynewdomain.com in the future.

How can I block search engines to enter this folder? Is it something that I can do easily? Blocking the access to this folder from the main domain will create some problems in the indexing of the site even when I will have a new domain?

Thank you a lot

in public_html/robots.txt:

User-agent: *
Disallow: /mynewdomain/
Disallow: /cgi-bin/
Disallow: /contact-
Disallow: /css/

This snippet prevents search engines from accessing the named folders, including the /mynewdomain/ folder, and anything that begins with "contact."

Place another robots.txt file in the mynewdomain folder to direct search engines on where not to go within your new domain once it goes live.

Google search robots.txt for more.

You can also check the referrer, if its a bot redirect to a page.

Originally Posted by L a r r y

in public_html/robots.txt:
Place another robots.txt file in the mynewdomain folder to direct search engines on where not to go within your new domain once it goes live.

Google search robots.txt for more.

What should I write in this robots.txt?

Originally Posted by webcosmo

You can also check the referrer, if its a bot redirect to a page.

How do I do that?

Originally Posted by pikus

What should I write in this robots.txt?

User-agent: *
Disallow: /newdomainfolder/

The first line in your robots.txt specifies the name of the user-agent, in this case, an asterisk, which specifies all robots, because you want the instructions to apply to all of them. Therefore, your first line must duplicate mine.
Second and following lines begin with the case-sensitive, "Disallow" and a colon.
Disallow is the only instruction you can place in the Robots.txt.
It will not allow access to whatever you specify.

You need to specify the name of the folder that houses the root of your new domain, substituting the name of your folder in the place of mine.

Add one Disallow: per line for each additional file or folder you don't want to allow to be included in Web search results. For me, that includes /cgi-bin/, /css/ or /stylesheets/, and other pages that I don't want to have appearing in the world's search engines.

Yes yes, thank you.
I understood how should be the robots.txt in the root folder.

But you said that I have to put another one on the same folder I want to exclude. And this is the one that I don't know what to write in and why I have to make another robots on the excluded folder, since search engines are already blocked by the robots.txt on the main folder?

Originally Posted by pikus

Yes yes, thank you.
I understood how should be the robots.txt in the root folder.

But you said that I have to put another one on the same folder I want to exclude. And this is the one that I don't know what to write in and why I have to make another robots on the excluded folder, since search engines are already blocked by the robots.txt on the main folder?

The newdomain folder's robots.txt is not needed at this time. . It will be at the web root of your new domain when the time comes for it to be activated. Sorry for this point of confusion!

thank you a lot! You gave me a good help!

can anyone tell me how to set the robots.txt on server?so the google can index all the content of my website?please help..thx before

and how if i want to block the google from indexing my expired webpage with the robots.txt...what is the command should i write in the robots.txt file,for example if my expired webpage is >www.myweb.com/1_10_Lenovo-ideapad-S10.html. so what should i type in the robots file?

Originally Posted by jacobsz

can anyone tell me how to set the robots.txt on server?so the google can index all the content of my website?please help..thx before

and how if i want to block the google from indexing my expired webpage with the robots.txt...what is the command should i write in the robots.txt file,for example if my expired webpage is >www.myweb.com/1_10_Lenovo-ideapad-S10.html. so what should i type in the robots file?

Robots.txt that allows full access to everything on the server is:

User-agent: *
Disallow:

The User-agent line applies to all robots, and an empty Disallow: will allow everything, and the presence of a robots.txt will eliminate the 404 errors from your error log when the expected robots.txt is not found. Having no robots.txt will have the same effect but with the error log reporting the missing file.

At least one Disallow: entry must be in the file, and this example meets the minimum requirement.

See: http://www.robotstxt.org/orig.html

You need to specify the name of the folder that houses the root of your new domain, substituting the name of your folder in the place of mine.