This is also an interesting idea:
http://serverfault.com/a/217066/46738
(accepted answer is pasted below)
Rate limiting login attempts is an easy way to prevent some of the high speed password guessing attacks. However, it's hard to limit distributed attacks and many run at a low pace over weeks or months. I personally prefer to avoid using automated response tools like fail2ban. And this is for two reasons:
1) Legitimate users sometimes forget their passwords. I don't want to ban legitimate users from my server, forcing me to manually enable their accounts again (or worse, try to figure out which of the 100/1000 banned IP addresses is theirs).
2) An IP address is not a good identifier for a user. If you have multiple users behind a single IP (for example, a school that runs NAT on 500 student machines) a single user making a few bad guesses can land you in a world of pain. At the same time the majority of the password guessing attempts I see are distributed.
Therefore I don't consider fail2ban (and similar automated response tools) a very good approach to securing a server against brute force attacks. A simple IPTables rules set to cut down on the log spam (which I have on most of my linux servers) is something like this:
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP
It prevents more than 4 connection attempts from a single IP to ssh in any 60 second period. The rest can be handled by ensuring passwords are reasonably strong. On high security servers forcing the users to use public key authentication is another way to stop guessing.
---- (end of answer)
for Ubuntu admins...
http://manpages.ubuntu.com/manpages/xenial/man8/ufw.8.html
(Uncomplicated Firewall) ufw supports connection rate limiting, which is useful for protecting
against brute-force login attacks. When a limit rule is used, ufw will
normally allow the connection but will deny connections if an IP
address attempts to initiate 6 or more connections within 30 seconds.
See http://www.debian-administration.org/articles/187 for details.
Typical usage is:
ufw limit ssh/tcp