How to set up a safe and secure Web server

Getting reports of brute force attacks is useful, but is not an indication of bad security. Hiding the reports under the carpet does not increase security either.

Preventing brute force attacks is not the bare minimum. The bare minimum is to not be vulnerable to brute force attacks in the first place. ssh already has built in protection for this, and it slows brute force attacks to the point where such an attack is impossible in practice, unless you have weak passwords. Mandate large enough keys only, ban passwords, and you're done. Allow password authentication is your real security vulnerability here. Patching over it with fail2ban just hides the issue.

I do understand how fail2ban works. "Just" monitoring logs isn't good enough. The data that appears in logs is not generally considered to be a security sensitive channel. It's string data with poorly defined delineation. It should not be trusted for automatic use, since every channel that dumps data into log files is not vetted for security.

> You've been lucky.

I disagree. You claim I'm advocating security by obscurity, but you're the one who seems to think that hiding version strings gains in security. That's security by obscurity, since you can determine the version of software by observing its behaviour (or just not caring and trying your attack anyway).

I'm not advocating complacency. We just disagree on what complacency is. I claim that if you keep your software up to date (easiest if you do follow distribution defaults), then you are sufficiently secure. The overwhelming majority of security compromises happen because people fail to run updates. The second largest cause is because of vulnerable misconfigurations that people have introduced. Only a tiny fraction of compromises come through a default distribution installation that your hardening would catch, and these holes are rapidly patched by vendors and a simple update will close them. I think that it is much more likely that you'll open the second cause (an introduced misconfiguration) if you try hardening and you don't know what you're doing. Which is why I recommend sticking to distribution defaults.

I think you overstate how much performance hit you get by running things virtually in ex. KVM or Xen. With the first, it's almost negligible, as you don't have to run a paravirtualized kernel in order to gain full access, so for Linux you are basically just limited by other factors, such as disk IO long before anything else start to kick in.

I don't buy your argument about latency one bit, do you have ANY data to back up your statement? You know - Google and many other big players run huge virtual machine clusters, you'd think they wouldn't if that almost unmeasurable difference in latency had an impact.

I would run KVM on top of a machine, even if there was only to be one VM on it, just because moving, backing up or scaling up or down can easily be done. You can also easily migrate the host to another physical machine if you are experiencing hardware issues, or just upgrading your rig. And yes, you can migrate a running server, I'm doing it all the time.

And yes, you can snapshot and take a backup of a running server, but what goes on inside of MySQL, or other applications for that matter, has nothing to do with the consistency of the disk image. When you snapshot a disk image, you naturally don't know whats in RAM.

If you don't know how to do it by hand, then puppet is just going to confuse you. Better to keep it simple when you're new at something.

"Getting reports of brute force attacks is useful, but is not an indication of bad security. Hiding the reports under the carpet does not increase security either."

I didn't say reports increase security, and Fail2ban does more than just reporting, it actively blocks brute force attacks. It's a bit difficult us discussing the merits of certain security measures when you keep focusing on the irrelevant as if those were my security suggestions - it's almost as if you're trying to 'death by a thousand paper cuts' my whole post simply to win an internet argument "Preventing brute force attacks is not the bare minimum. The bare minimum is to not be vulnerable to brute force attacks in the first place. ssh already has built in protection for this, and it slows brute force attacks to the point where such an attack is impossible in practice, unless you have weak passwords. Mandate large enough keys only, ban passwords, and you're done. Allow password authentication is your real security vulnerability here. Patching over it with fail2ban just hides the issue."

For SSH, you'd be right. But as I've repeatedly said, not all services offer key based log ins and sometimes there's a business requirement for password log ins on systems that could be managed with keys. You're original arguments were about usability yet the arguments you're making now are the lest flexible suggestions raised thus far!

"I do understand how fail2ban works. "Just" monitoring logs isn't good enough. The data that appears in logs is not generally considered to be a security sensitive channel. It's string data with poorly defined delineation. It should not be trusted for automatic use, since every channel that dumps data into log files is not vetted for security."

Logs are fine for parsing as you have to be compromise before the logs are comprimised. But which point, it's already too late.

"I disagree. You claim I'm advocating security by obscurity, but you're the one who seems to think that hiding version strings gains in security. That's security by obscurity, since you can determine the version of software by observing its behaviour (or just not caring and trying your attack anyway)."

In theory I'd agree with you, however a great number of compromised systems were attacked by opportunists scanning version numbers looking for boxes to target with known vulnerabilities. Plus, and once again I'm having to repeat myself, those specific changes I advised are actually required to comply with many compliance laws (eg PCI compliance, required if you make finance transactions in the UK).

"I'm not advocating complacency. We just disagree on what complacency is. I claim that if you keep your software up to date (easiest if you do follow distribution defaults), then you are sufficiently secure. The overwhelming majority of security compromises happen because people fail to run updates."

There's no such thing as 'sufficiently secure' as that only depends on the attackers targeting your system. Today you might be 'sufficiently secure' because your box has not been spotted by any keen attackers, tomorrow might be different.

Also, I'd be more inclined to agree with you if all of the examples you've given weren't off topic from the points I raised or just incorrect (eg changing Apache config being dangerous and/or hard, fail2ban reducing usability, etc).

"The second largest cause is because of vulnerable misconfigurations that people have introduced."

I'd go along with that. I've often said 'users are the biggest security risks' :)

"Only a tiny fraction of compromises come through a default distribution installation that your hardening would catch, and these holes are rapidly patched by vendors and a simple update will close them."

None of the configurations I mentioned (bar the list of paranoid ones) fall into that category; non-optimal configuration isn't a hole that gets patched. Plus even if it was, it wouldn't be fixed with software updates as package managers tend to avoid over-writing live config files else they'd risk doing more damage than good.

"I think that it is much more likely that you'll open the second cause (an introduced misconfiguration) if you try hardening and you don't know what you're doing. Which is why I recommend sticking to distribution defaults."

You can't make a system less secure by changing the settings I recommended as the distro defaults are already on the most open defaults. Plus, and once again I'm repeating myself, the configurations I'm recommending are incredibly easy to implement.

I find it odd that we're actually arguing about whether it's worth making the most basic of changes based on the assumption that those people in question are stupid and their box will probably be ok. Surely a better approach would be to suggest optimisations; guiding them through the process if needs be? After all, it's too late to regret using the defaults if and when you get hacked (and in my line of work, I've had to fix quite a number of boxes where the sys admins have been content just running with the default settings).

This sort of feels like an argument with someone who prefers to optimize the hell out of things in the best possible way. Thats a fair position to have and to take in this day and age.

What virtualization allows you to do is optimize the organizational responsibility. If I am developing internally on my VM, when it gets certified by ops management, and rolled out onto the live VM stack, I can be very sure what I am being responsible for; the ops also can be sure (because they maintain a huge VM filesystem, mostly) that they don't have to deal with things at a very thin slice.

For web operations, look, its simple: VM is good enough because there is so much overhead all over the typical web stack that a few frames of difference are, largely, irrelevant to the use case. Okay, don't put your multiplayer live realtime universe hashes in a VM; those belong bare metal. But the web front ends that are serving the same old content, over and again .. these are well worth putting in a package which can be massively deployed at whim.

A typically well-packed VM, consisting of only the hand-optimized built image of a development team with this in mind, can be a very, very tight package. I've seen reflectors and mongodb proxies and so on, packed into 256meg boot image that can be replicated simply by giving it a new name .. deploy 2,000 copies of these VM's, and you have a massive solution to the front-door problem..