Most 404 accesses are from people probing a server for common URL's belonging to vulnerable web applications. Thus my 404 page is a honeypot, and too many 404 accesses in a short period results in an automatic IP ban.
Include a directory "visiting-this-will-ban-you" in your robots.txt that IP bans whomever visits it.