Hi all,

One of my website is getting problem with some crawler. they are hitting contiously by http request, I have my own server that is getting extra load.

one of the crawler of www.openwebspider[dot]org, is creating more problem, I have just block this site from firewall. but can u suggest why this problem is going on again & again, Is there possibility someone is hacking my website, I have linux installed server.

please suggest.


Thanks.

It's not someone hacking your site; just indexing it. Maybe looking for some jpegs or something. The same bot ( and two others at the same time ) brought my blog down for excessive cpu usage. It's pretty much what the name says; you can download the source code and run your own bot.

Try this in your .htaccess file:

RewriteCond %{HTTP_USER_AGENT} ^curl [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^OpenWebSpider [OR]
...
RewriteRule ^(.*)$ http://www.robotstxt.org/