Ask HN: How do you handle server security/intrusion monitoring?

Set up the server in a secure way, with many layers of defense. Any applications that accept data from the internet should be very robust. After that, add mitigating stuff like SQL permissions, a chroot jail for the web server, running as a totally unprivileged user (i.e. not nobody), and so on and so forth. Pick a secure kernel, too, and make sure that your software is compiled with exploit mitigation techniques (e.g. ASLR, W^X). Throw out anything that needs patching more often than once a year (once every ten years is a better goal). Don't pick a weak password and leave SSH password logins enabled.

If you are interested in detecting intrusions, a first layer may be a list of checksums of crucial files. This is built-in to my favourite OS, OpenBSD, but others can use Tripwire, AIDE or any of the other programs. Note that an attacker with root privileges can typically bypass this.

EDIT: Look at mod_security, too, if you use Apache and cannot trust the web developer (which may be you). Just accept that it will, in the end, not save you.

EDIT2: And don't store passwords in plain text. Or MD5.

rkhunter and chkrootkit will both look for insecure configuration, permissions, and known root kits. rkhunter will also maintain a database of the signatures of your system binaries. I assume there are overlaps between the two. I don't really care, I run both.

logcheck will watch your logs and email you when something shows up in a log that doesn't match a configurable set of regular expressions. When I add a new piece of software to a system, logcheck goes berserk. I go through the emails I receive and add in new regex's to ignore the harmless notifications until they finally taper out and disappear, only leaving me with the true exceptions.

Look over the documentation for all three of these, and make sure that they're actually running. Personally, I don't want an email unless something is wrong. If I get an email every day, I'll start to ignore it. I don't think I'd notice if an email I was ignoring didn't show up, so I optimize for no output unless it's something I should read. I chase down the cause of every exception in my logs. No excuses. If it isn't an exception, I add a regex to logcheck.

I run monit because it is pretty easy to set up, and it sends me an email when a server process gets changed or restarted. It lets me enforce limits on a server process, so I can potentially get warning of an attack as it happening. I suspect monit is nowhere near the most sophisticated tool for this purpose, but it meets my needs very well.

As for other basic security stuff... Don't use port 22 for ssh, use something much higher so that you won't get the continuous pounding on the front door that an ssh server on port 22 receives. Disable root login, as well as password login. Use certificates to log in to your machine. Configure servers to listen to 127.0.0.1 whenever you can. Install a software firewall to only allow access to http and ssh anyway.

Unless you've done something stupid with your web app, you've made it incredibly difficult to impossible for script kiddies. If you piss off an elite black-hat, it's probably safe to assume they can own you, and you likely won't know it. So make good backups, with regular offline backups that can't get destroyed by a hacker with a particularly malicious bent.

2 - Install things that you understand. I don't install large meta-packages. I also install most things from source: nginx, ruby, gems, postgresql. All these things make/install without much effort. When compiling, only include options you need.

3 - Lock down SSH. Login only with SSH keys, protocol 2. Also, change your sshd port to something less likely to be scanned. This way, your SSH log files won't have too much garbage in them and you actually can check them with little effort. You can find lots of tutorials on locking down ssh.

4 - Document _everything_. If all else fails, your docs enable you to recreate your server environment within a few hours.

If you do all this correct, when you upgrade your server, e.g.: apt-get upgrade, you will only be upgrading basic distro stuff, mostly security updates.

oh yeah, _little_things_ like making sure your ruby/blub app processes are connecting to your db with an "app" id instead of an "admin" one is a choice decision ;).

Seriously, it puts rkhunter, logcheck and all these old tools in their toes..

Check http://www.ossec.net

It monitors your logs, file changes, etc all by default, simple to install, etc.. I love it.