How to set up a safe and secure Web server

1) Changes that give you relatively little gain from a security point of view at the cost of massive usability. I'd be happy to run production servers without these changes, and so I claim that these are debatable.

2) Changes that stop inexperienced admins accidentally compromising their servers, at the cost of usability/complexity. Debatable because they're less about security and more about not shooting yourself in the foot. Eg. fail2ban. I use ssh keys only. Brute forcing my ssh won't work, and I'd prefer to not allow outsiders to cause firewall rule changes on my servers based on a script that parses log strings with regular expressions. I trust ssh's security more than I trust fail2ban in not having a vulnerability.

> And lastly, if you're really paranoid, run your webserver inside a Linux/UNIX container

This is pretty much what AppArmor hardening does. LXC (Linux containers) are implmented via AppArmor. A compromised httpd daemon restricted in this way won't be able to much anyway (eg. open outbound ports or go to other areas of the filesystem). And this is what Ubuntu ships by default for many daemons.

> Also many distributions ship FTP, which is insecure by default.

Nowadays this stuff is only installed if you requested it, which is different from "ship"ing it. If you use distribution defaults, you won't have an FTP server installed. If you choose to install one, you'll get the weakness whether you use the distribution, tune it yourself or install from a third party source.

> In short; distribution's don't ship secure defaults. They ship a happy medium between usability and security. However if you have an internet facing box (particularly one not hidden behind a hardware firewall, as many budget set ups are not), then the happy medium isn't secure enough.

Secure enough for production. I have plenty of servers on distribution defaults that haven't been compromised (and I have experience in dealing with others' servers when they have been compromised, so I don't think I'm oblivious). You may prefer adding additional hardening, but it is unnecessary and relies on you knowing what you are doing.

Finally, you've missed my biggest point. For someone new to running servers on the Internet, who can he trust for guidance? You, me, a random guide on the Internet or a distribution vendor?

The first two paragraphs are about one benefit: using otherwise idle resources. At my lab, we have five powerful servers for data-processing experiments and non-time-critical analytics. Most of the time they sit completely unused. By installing Xen and running things in virtual machines, we have been able to also put tens of differently configured web servers for various small projects on each.

The third paragraph is also very valuable from experience. It has been very useful to keep documentation around how different servers are set up, and verifying that the actual setup still matches the documentation. This has been easiest to ensure by having scripts that create a new VM, install packages and make configuration changes - if in doubt, completely delete the VM and re-create it from scratch with a single command-line.

If you don't mind, I will address your points individually:

"Changes that give you relatively little gain from a security point of view at the cost of massive usability. I'd be happy to run production servers without these changes, and so I claim that these are debatable."

Which items specifically prevents usability? None of them do aside the FTP (but SFTP is supported by nearly every FTP client already) and turning off PHP error logging to STDOUT (really not recommended for production servers!). Everything else doesn't reduce usability for authorised users. And I resent the claim that there's little gain as even on my servers which have a low public profile, I get reports of multiple brute force attacks a week (on average, about separate 5 attacks a week, all of which I manually add to a permanent firewall blacklist when it becomes obvious that they're just retrying attacks everytime fail2ban's autoban expires).

"Changes that stop inexperienced admins accidentally compromising their servers, at the cost of usability/complexity. Debatable because they're less about security and more about not shooting yourself in the foot. Eg. fail2ban. I use ssh keys only. Brute forcing my ssh won't work, and I'd prefer to not allow outsiders to cause firewall rule changes on my servers based on a script that parses log strings with regular expressions. I trust ssh's security more than I trust fail2ban in not having a vulnerability."

You don't understand how fail2ban works if that's your primary concern. Fail2ban isn't public facing, it just monitors logs and then adds an iptables (or any firewall you chose) rules based on the results of the log files. It's iptables that is public facing and thus needs to be protected against vunrabilities; and iptables is proven technology already. What's more, preventing brute force attacks doesn't reduce usability nor prevent you from shooting yourself in the foot; preventing brute force attacks is the !!bare minimum!! you need to do if you have an internet facing server. This is security 101! I do agree with you that SSH keys are preferable to password log ins, but that also adds complexity and can reduce usability if users need access from multiple locations and platforms (there are workarounds, eg memory stick with the key on), and then you still have the issue of brute force attacks on every other log in service.

"This is pretty much what AppArmor hardening does. LXC (Linux containers) are implmented via AppArmor. A compromised httpd daemon restricted in this way won't be able to much anyway (eg. open outbound ports or go to other areas of the filesystem). And this is what Ubuntu ships by default for many daemons."

You missed the point about easy snapshotting though, which is why I raised the point about containers in the 1st place. I will grant you that you can have back up solutions on bare metal systems, but snapshotting is way more usable (which is ironic given that's been your key argument). Plus I did say that advice was for the paranoid (ie not really all that necessary unless you fancy tinkering).

"Nowadays this stuff is only installed if you requested it, which is different from "ship"ing it. If you use distribution defaults, you won't have an FTP server installed. If you choose to install one, you'll get the weakness whether you use the distribution, tune it yourself or install from a third party source."

You're just reiterating what I said though as I didn't say distributions install it by default. However you missed my point there as well, as I was describing how FTP isn't ideal for production systems (FTP sends clear text passwords and doesn't behave itself behind firewalls and/or NATing without adaptive routing (which means that FTPS often doesn't work behind some firewalls).). Thus chrooted SFTP is a much saner solution for production systems.

"Secure enough for production. I have plenty of servers on distribution defaults that haven't been compromised (and I have experience in dealing with others' servers when they have been compromised, so I don't think I'm oblivious). You may prefer adding additional hardening, but it is unnecessary and relies on you knowing what you are doing."

You've been lucky. But what your advocating is little more than 'security by obscurity' rather than pro-actively hardening your box. Having worked on a number of high profile web infrastructures (including farms that take web payments), I wouldn't be doing my job if I ignored my steps; in fact our business wouldn't exist as we would break UK laws.

The Apache configuration I described is ridiculously easy to set up, as is fail2ban. They are the bare minimum any web server should implement. Which is why I placed emphasis on them by placing those suggestions at the top of my post. However if we're going to advocate complacency in order to make our lives easy, then lets also not bother keeping our software up to date, because that's a real pain in the arse at times :p

A harddisk seek + reading 1 MB sequentially is something like 30 times slower than SSD and 120 times slower than reading from RAM. Disk seeks are what really kills you, as a disk seek is 20 times slower than a roundtrip within the same datacenter.

Sending 1 MB of data over a 1Gbps network using a disk seek and a sequential read is 3.6 times slower than doing the same with SSD. For big files with a server that has to serve many concurrent requests you'll end up doing several disk seeks for the same file. And we've been talking about the ideal too, because data on disks can get fragmented and so reading from the same file does not guarantee a sequential read.

This is why Varnish does caching, even when serving static files. If you want high performance with low latency, then Varnish works better than something like Nginx.

As an example, right now I'm working on integrating with an OpenRTB ads exchange platform. My server has to respond within 100ms total roundtrip (with an upper bound of 200ms, but they complain if you consistently respond in over 100ms). It's enough to say that everything to be served has to be already in memory, as doing anything else means that window cannot be met (consider how a single disk seek can cost something like 10ms).

So yeah, SSDs can be useful.