Unfashionably secure: why we use isolated VMs

This is nice in its uniformity (same tool works for any distro that has an existing AMI to work with), but it's insanely slow compared to just putting a rootfs together and uploading it as an image.

I think I'd usually rather just use whatever distro-specific tools for putting together a li'l chroot (e.g., debootstrap, pacstrap, whatever) and building a suitable rootfs in there, then finish it up with amazon-ec2-ami-tools or euca2ools or whatever and upload directly. The pace of iteration with Packer is just really painful for me.

Also to add, we already have lots of customers who use this model.

I agree. VMs rely on old technologies, and are reliable in that way. By contrast, the move to Docker then necessitated additional technologies, such as Kubernetes, and Kubernetes brought an avalanche of new technologies to help manage Docker/Kubernetes. I am wary of any technology that in theory should make things simpler but in fact draws you down a path that requires you to learn a dozen new technologies. The Docker/Kubernetes path also drove up costs, especially the cost associated with the time needed to set up the devops correctly. Anything that takes time costs money. When I was at Averon the CEO insisted on absolutely perfect reliability and therefore flawless devops, so we hired a great devops guy to help us get setup, but he needed several weeks to set everything up, and his hourly rate was expensive. We could have just "push some code to a server" and we would have saved $40,000. When I consult with early stage startups, and they worry about the cost of devops, I point out that we can start simply, by pushing some code to a server, as if this was still 2001, and we can proceed slowly and incrementally from there. While Docker/Kubernetes offers infinite scalability, I warn entrepreneurs that their first concern should be keeping things simple and therefore low cost. And then the next step is to introduce VMs, and then use something like Packer to enable the VMs to be uses as AMIs and so allow the devops to develop to the point of using Terraform -- but all of that can wait till the product actually gains some traction.

I like to say that Docker wouldn’t exist if the Python packaging and dependency management system weren’t complete garbage. You can draw a straight line from “run Python” to dotCloud to Docker.

Does that jive with your experience/memory at all? How much of your motivation for writing Docker could have been avoided if there were a sane way to compile a Python application into a single binary?

It’s funny, this era of dotCloud type IaaS providers kind of disappeared for a while, only to be semi-revived by the likes of Vercel (who, incidentally, moved away from a generic platform for running containers, in favor of focusing on one specific language runtime). But its legacy is containerization. And it’s kind of hard to imagine the world without containers now (for better or worse).

1. Commercially motivated attacks against mere mid-sized entities (100 M$-1 G$ in revenue) can empirically derive 10s of M$ from successful attacks. Adequate security must make such attacks unprofitable. This constitutes a absolute, rock-bottom, minimum standard for effective enterprise security.

2. We have always known that such attacks would eventually become feasible to execute once the hackers matured. This was so obvious that government security standards/certifications such as the Rainbow Series codified in the 80s already considered such threats and placed thwarting them as the middle-levels of security. So, we have always known that "adequate" security against commercially motivated attackers has always demanded protection against skilled teams.

3. The commercial IT systems in regular use have never once, over 40 years, demonstrated the levels of security needed to protect against such commercially motivated attacks. These techniques and processes have failed continuously to achieve adequate security despite claiming to solve it every year for 40 years. At some point you need to stop listening.

4. There are systems that did achieve adequate security against commercially motivated attackers and even security against state actors in those 40 years. These standards were not pie-in-the-sky unreachable goals. They were "practical" if you actually cared about security.

5. KVM is in the commercial IT system category, being derived and developed by people who have never once deployed, developed, designed, or likely even seen a system known to have adequate security. Such systems DO NOT get the benefit of the doubt. There is no metric of evaluation, no means of evaluating if the theorized improvements achieve adequacy. In fact, you would be hard-pressed to find literally anybody who would stand up and say: "KVM is unhackable by any team with a budget of 10 M$" (budget including the average salary for the team members so you do not get a talented team doing it just to prove it can be done). Nobody will vouch for the system claiming it achieves even the bare-minimum requirements for adequate security I stated above. That it might be "better" than the Linux kernel is irrelevant; bad is also better than terrible, but it is still bad. At the end of the day they are not meaningfully different; they are all inadequate and unfit for purpose.

So, you can disagree on two primary positions:

1. 10 M$ is too high of a standard for mid-size enterprise security.

2. Commercial IT systems, such as KVM, achieve the 10 M$ level.

You could also argue that I am being reductive by defining security as the cost for a successful attack. But that is silly because it is the one metric that exactly aligns with the operational goals. Every other metric is a means of helping quantify the cost of a successful attack. To put it another way, if you had a magic genie that told you that number, you would not even bother with any other metrics; you have a direct line to what matters.

In summary, it is none of the above. The betterness of KVM is irrelevant except if it achieves adequacy. There is no evidence of that and until there is, KVM is not categorically distinct from any other inadequate system. So, the original point still holds, the reason these "secure VM" techniques are not applied to make a "secure OS" is that there are no "secure VM" techniques to be found in that corner of the world.