Block web scanners with ipset and iptables
1–10 of 46 posts
Re: Block web scanners with ipset and iptables
#2I understand the first part -- sending requests with no host header to a spam log (or even better, don't log).
What I don't understand is the second part -- blocking those hosts. Seems pointless now that you've de-noised your logs. They're still sending packets. Saves thousands of bytes on outbound?
What about all the scan-spam on sites WITH host headers? Whatevs.
Re: Block web scanners with ipset and iptables
#3I understand the first part -- sending requests with no host header to a spam log (or even better, don't log). What I don't understand is the second part -- blocking those hosts. Seems pointless now that you've de-noised your logs. They're still sending packets. Saves thousands of bytes on outbound? What about all the scan-spam on sites WITH host headers? Whatevs.
Serves a few purposes, but as you said the main objective is already done by de-noising.
The other reason I do this is because it's easy to detect that kind of scanning in HTTP logs, but not as easy for other services (ssh, ftp, smtpd, etc) without something like fail2ban, and the blanket ban applies to all of them. So, if a bot scans your HTTP server enough times, they can't go after "softer" targets later.
For scan-spam that does hit your "real" site, it's a bit more tricky as there absolutely will be false positives. You can grep for all 401/403's and add them to the list, but that will sooner or later hit a real user. So it's much more specific to the application you're hosting, where this works for just about any site. The other nice thing is that even when they scan your "real" site, they'll often hit the default host via IP scans at the same time, so you can still manage to ban them.
It's not perfect, but it's good enough :)
Re: Block web scanners with ipset and iptables
#4Another neat trick is to add a link in robots.txt and instruct bots to stay away. If they don’t, you add them to your blocklist
Re: Block web scanners with ipset and iptables
#5I understand the first part -- sending requests with no host header to a spam log (or even better, don't log). What I don't understand is the second part -- blocking those hosts. Seems pointless now that you've de-noised your logs. They're still sending packets. Saves thousands of bytes on outbound? What about all the scan-spam on sites WITH host headers? Whatevs.
Serves a few purposes, but as you said the main objective is already done by de-noising. The other reason I do this is because it's easy to detect that kind of scanning in HTTP logs, but not as easy for other services (ssh, ftp, smtpd, etc) without something like fail2ban, and the blanket ban applies to all of them. So, if a bot scans your HTTP server enough times, they can't go after "softer" targets later. For scan…
Running your own mail server is a great source of data to identify botnet-compromised hosts.
When I started banning IPs that send "HELO " for 24 hours, I cut the number of fake login/registration attempts on a bunch of my web-based projects by ~50%.
It works the other way, too. Temporary bans on hosts that try to access /wp-admin (I don't run Wordpress anywhere) cut my email spam significantly.
(Some day, I'll get around to implementing a real reputation tracking system, with exponential ban lengths.)
Re: Block web scanners with ipset and iptables
#6On my internet facing hosts, I use the firehol level 2 and level 3 block sets along with blocking all CN IP space that I can accurately identify. My logs are eerily quiet.
Re: Block web scanners with ipset and iptables
#7Earlier quoted context omitted.
Serves a few purposes, but as you said the main objective is already done by de-noising. The other reason I do this is because it's easy to detect that kind of scanning in HTTP logs, but not as easy for other services (ssh, ftp, smtpd, etc) without something like fail2ban, and the blanket ban applies to all of them. So, if a bot scans your HTTP server enough times, they can't go after "softer" targets later. For scan…
Running your own mail server is a great source of data to identify botnet-compromised hosts. When I started banning IPs that send "HELO " for 24 hours, I cut the number of fake login/registration attempts on a bunch of my web-based projects by ~50%. It works the other way, too. Temporary bans on hosts that try to access /wp-admin (I don't run Wordpress anywhere) cut my email spam significantly. (Some day, I'll get ar…
This is an important aspect of it - you can use information on one angle of attack to protect other devices.
Do note that doing this kind of thing can block people on Tor, because Tor is used for attacks quite often, also.
Re: Block web scanners with ipset and iptables
#8Actually you don't need to respond to bogus http clients at all: https://gist.github.com/radupotop/2aef0bdc0ccbd3a706044e3598...
Re: Block web scanners with ipset and iptables
#9I wonder why the author uses a 404 error response. I usually configure NGINX with "return 444;" which closes the connection without response. Scanners don't deserve a response. I may have wasted bytes receiving the request, but I won't waste any more once I know the request is garbage.
Re: Block web scanners with ipset and iptables
#10Another neat trick is to add a link in robots.txt and instruct bots to stay away. If they don’t, you add them to your blocklist
I was confused at what you were saying at first. For those that may also be confused:
You can add something like this to your robots.txt:
User-agent: *
Disallow: /some/unguessible/url