I use a program (Guardian) to send me notifications when an error occurs on my site (404, 403 etc.) These notices contain the URL causing the error and the referring URL. Lately I have been getting numerous notifications with URLs to directories without index files (On a side note, these URLs also contain within them, urls to other sites. I assume this is some spammer technique and I am not too concerned about this in this post but if someone can shed some light on how/why they (the spammers) do this I would appreciate it.)

My purpose in posting here is to find out about the use of index files in directories. I have read that this is a good idea, but I have yet to hear why except that in case someone navigates to the URL of the directory rather than the actual page URL, then they will not get a 403 error message, but this seems like a rather weak reason to me. Is there another reason? SEO reason? Technical reason?

If I find good reason to do this, is it enough to just put a blank index file in each directory? If so why? What purpose does it serve (aside from thwarting the spammers doing what I described above.) Thanks in advance for your help.

It's not so much spammers that it stops, but it can prevent "crackers" from "seeing" what files are in the folder. So if the "hunting" bot can't "see" them it won't report a potential target for an attack.

Originally Posted by chrishirst

It's not so much spammers that it stops, but it can prevent "crackers" from "seeing" what files are in the folder. So if the "hunting" bot can't "see" them it won't report a potential target for an attack.

They can't see what's there if the server is set to respond with a 403 error page, which is what I have done as I indicated above.

From a security standpoint best practice states that you should include an index for the reason chris outlines above. It's a belt and braces approach really, if something happens on your webserver and it suddenly allows directory listings as a result of mis-configuration then index files will add an extra layer to prevent directory listings.

Originally Posted by Monkey Do

From a security standpoint best practice states that you should include an index for the reason chris outlines above. It's a belt and braces approach really, if something happens on your webserver and it suddenly allows directory listings as a result of mis-configuration then index files will add an extra layer to prevent directory listings.

That's a clear and good reason. Now I am convinced. So, just a blank index.htm file is all I need?

Yep. ..

Or, if apache, you could deactivate the directory listing at all.
It would prevent it without needing to put a blank file everywhere.
http://httpd.apache.org/docs/2.0/mod/core.html#options

Simply add a section like this:

<directory />
  options -indexes
</directory>

in a .htaccess in the root directory would disable server generation of a list of files in any directory. A 401 - forbidden will be returned.

Warning: the default value of options, if not specified, is "all", so by adding just -indexes, you deactivate all the other.
You must add them (with a +name_of_option) to enabled them back.
I would usually use this:

options -indexes +ExecCGI +FollowSymLinks +Includes

The reason to have an index.html file is to prevent directory contents from being provided to surfers when they request the html file specifically. The problem is that in most cases, putting a blank index.html file in the directory overrides the index.php call. This is not good if it is a WordPress or forum directory.

To override this use something like this in the directorys .htaccess file:

DirectoryIndex index.php index.html index.htm index.cgi
Options -Indexes

This makes the index.php file the default and insures no one can see the contents of your directory. Setting indexes off-- does not always fix the problem.

Q...