Welcome to Server Circle. It's a friendly site and all levels of experience are welcome. Be aware that we use cookies for your login.
Server Circle - Ask questions about Servers and get answers from experts.
Beta (0.36 sec)
Evil bots and spiders

Are there any lists of which ones I should ban ?

My web logs are so full of stuff these days it's hard to know what is legitimate traffic and what isn't.

Thank you.

Asked by:
TomPating
1355 points
 Report Abuse
 Share Page - Category: CentOS Linux - Tags: Evil bots and spiders
 Enter your response
Please use Pastie.org to paste lengthy code or to fix formatting issues with code
  • Responses in reverse (2)

Blocking requests by user agent might stop some of these getting through but there is no reason what a malicious scanner cannot set this to the same value as a legitimate browser. Can you not filter your logs with grep to contain requests only for valid resources? Filter out 404 errors and you'll get rid of a lot of the junk.

Response by:
Jimbob
437 points
A bot trap is a good starting point:

http://danielwebb.us/software/bot-trap/

Response by:
stevenrossuk
1325 points


  • Related Questions


by poppy (1375 points) in CentOS Linux - 0 replies



About Us : Contact Us : Etiquette : Terms : CDN Failover : ShorterURL : CDN Fallback : © 2013 Server Circle