Internet Marketing Tycoon

**james77** · 05-11-2012, 07:57 AM

Is robots.txt necessary for all the websites? If I not created and submit for my website what will happen? Is that affect my website ranking or indexing?

**chrishirst** · 05-11-2012, 03:06 PM

No, Nothing at all & No in that order.

**TWD** · 05-11-2012, 08:07 PM

It's not necessary but its sometimes a good idea, depending on how your content is organized, in order to prevent duplicate content indexing by google.

See the matt cutts video
http://www.youtube.com/watch?v=mY9h3G8Lv4k

**diyakapoor** · 05-14-2012, 08:25 AM

No there is nothing which will affect your rankings or SEO. robots.txt file is only to stop search engines access on some particular pages of website which you don't want to crawl. So, if you don't have any page like this no need to create robots.txt file

**aruldave** · 05-14-2012, 09:25 AM

Only the legitimate search engine crawlers follow the robots.txt others they don't or no need to.

**Sabrina002** · 05-18-2012, 05:14 AM

Not necessary for website. As our website doesn't have it. It depends on the content of your page.

**kevenlee** · 06-05-2012, 06:27 AM

Quote:

Originally Posted by james77

Is robots.txt necessary for all the websites? If I not created and submit for my website what will happen? Is that affect my website ranking or indexing?

its not necessary that you have to create a robots.txt file for your website, it is require only when you don't want to index certain category of the website or certain pages like admin, lo gin and registration pages. this file instruct search engine spider to index and do not index pages or folders which you specify. If you are not writing any robots.txt file then by default all pages of your website will index into the search engine.

**syedakram** · 06-07-2012, 12:10 PM

Quote:

Originally Posted by james77

Is robots.txt necessary for all the websites? If I not created and submit for my website what will happen? Is that affect my website ranking or indexing?

No, it is not compulsory to submit the robot.txt,
Search engines use the robot.txt file to fully crawl the website not to rank the websites!

**chrishirst** · 06-07-2012, 02:18 PM

Quote:

Search engines use the robot.txt file to fully crawl the website

No they don't.

**chrishirst** · 06-13-2012, 01:27 PM

Quote:

Originally Posted by enlightentechno

Robots.txt helps You to allow/disallow and control your Site Links to get Crawled.

Nope!!

It can ONLY tell compliant "robots" where they cannot go.

**enlightentechno** · 06-13-2012, 04:21 PM

Quote:

Originally Posted by chrishirst

Nope!!

It can ONLY tell compliant "robots" where they cannot go.

Ok, so does it helps to stop bot to crawl a page?

**chrishirst** · 06-13-2012, 05:31 PM

If that question is

Does it stop "bots" from crawling URLs?

The answer is, Yes provided that "bot" reads and uses the disallow instruction.

**infoworldelite** · 06-14-2012, 05:43 AM

Robots.txt file is a text file that inform search crawlers which page you like them not to index. For example, if you want to keep them from indexing everything under your private directory, you would include a Disallow: /private/ field. For every more information about robots.txt, check out this guide: a robots.txt file guide that won't put you to sleep.

**ryangetty** · 06-28-2012, 06:32 AM

No, its not necessary but if someone use it, it helps him to tell the search engine that which page or file to be crawl or which not.

**Benmoore47** · 07-04-2012, 10:35 AM

some use that as a reason in site's KW ranking decrease.

**jasonmilsont** · 07-07-2012, 02:41 PM

Robot.txt is the one of the google web master tool by which you can hide some data or pages of your site from the google crawler which you do not have to need to crawl.

**chrishirst** · 07-07-2012, 03:46 PM

Quote:

Originally Posted by jasonmilsont

Robot.txt is the one of the google web master tool by which you can hide some data or pages of your site from the google crawler which you do not have to need to crawl.

No, No, NO!!!!