Reply
Old 06-24-2009, 10:43 PM Search Spiders
Experienced Talker

Posts: 30
Trades: 0
I am in the process of setting up my first website and wanted to start uploading my pages to a server soon. However, I wanted to test them on the server and make sure that everyone involved with the project was happy before any search engines crawl the page.

My question is, if I don't submit the page to Google, Yahoo and other search engines do I need to worry about them crawling my page or do I need to setup an htcacess file to keep them at bay?
samo1215 is offline
Reply With Quote
View Public Profile
 
 
When You Register, These Ads Go Away!
Old 06-25-2009, 12:47 PM Re: Search Spiders
Skilled Talker

Posts: 55
Trades: 0
If no one links to you and you don't tell the search engines then no one knows you exist so test away.
Scarthingwell is offline
Reply With Quote
View Public Profile Visit Scarthingwell's homepage!
 
Old 06-25-2009, 05:34 PM Re: Search Spiders
hairygunther's Avatar
Extreme Talker

Posts: 164
Trades: 0
Quote:
do I need to worry about them crawling my page or do I need to setup an htcacess file to keep them at bay?
To be safe, set them up in folders just for testing - then move the files to the main area. After all, there's various ways google can find a link to your site, eg your own carelessness. HTaccess is a bad idea because then if it gets crawled google has it pegged as blocked, and you have to wait for it to recrawl and who knows whether you'll get unlucky about the time delay. Leaving it as it is and taking a risk is also bad, simply because it is careless, unprofessional and shoddy.

This is a trivial question worth far less ascii than I have dedicated to it. I couldn't ever see myself ever having asked such trivial questions anywhere on my journey to development expertise... I think you need to try and rely more on your mind and more on your own drive and less on the (often incorrect) advice of those in an arena (social media / consumer generated media) with over 75% deceptive messages, written with the primary purpose of selling in mind.
hairygunther is offline
Reply With Quote
View Public Profile
 
Old 06-26-2009, 04:32 PM Re: Search Spiders
konetch's Avatar
Ultra Talker

Posts: 252
Name: Alex
Location: Michigan
Trades: 0
even if you're careless it will probably take google more than a week to index your site, so if you can get everything done within that time you won't have a problem
__________________
Alex
konetch is offline
Reply With Quote
View Public Profile Visit konetch's homepage!
 
Old 06-28-2009, 01:11 PM Re: Search Spiders
Skilled Talker

Posts: 57
Name: Larry K
Trades: 0
Robots.txt should keep the spiders away from the test areas, and then just transfer agreed-upon content from the test folders into the final resting place on the site.
L a r r y is offline
Reply With Quote
View Public Profile
 
Reply     « Reply to Search Spiders
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML

 



Page generated in 0.12098 seconds with 13 queries