Having google spider each page? and add links

**HaRRo** · 05-30-2005, 02:43 PM

Hey all i have started a few forums and modded them to html files.

For google to add each page to its search engine have i to add anything in robots.txt or any other mods?

Thanks for any suggestions

**zilla** · 05-30-2005, 02:59 PM

Well it will be good to exclude your scritp files from indexing cos google doesn't like duplicate content.

**HaRRo** · 05-30-2005, 03:13 PM

what script files?

**HaRRo** · 05-30-2005, 03:14 PM

all forum files have been modded to be html , with title i.e forum-title-text.html on each forum etc... using mod rewrite