Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Execute Docker Containers as QEMU MicroVMs

mergeboard.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: Execute Docker Containers as QEMU MicroVMs

#31

Post by handrous »

coder543 wrote: post #30
handrous wrote: post #26

Earlier quoted context omitted.

> > the single important binary > This is also such a weird thing to throw out there. I like a good Go program myself, but most companies are not only deploying single-binary statically linked applications. Most companies are also deploying some kind of Ruby, Python, or Java application... none of which are likely to be a single file in practice. Sure, but usual practice with containers is to put each thing in its ow…

> Sure, but usual practice with containers is to put each thing in its own, unless they are very tightly coupled. Web-app with a SQL database and a memory cache? Three containers. You can do otherwise, but that's typical. Usually each container ends up with one main, important running process, and not much else. I agree, but... getting all the application dependencies in there is more than just getting a single binar…

My personal favorite thing about Docker, and the part I'd most miss if I switched to Jails (which I'm fairly confident could meet my needs with some fairly simple scripts and aliases that wouldn't take me long to arrive at, which is why I think there's so much less of an "ecosystem" there, even a nascent and under-developed one) is the way it forces projects to un-fuck their configuration.

500-line config, much of which few people ever care about, with all kinds of ill-conceived nesting? Better put the ~20 options that 99% of users ever touch in environment variables, and document them. Weird state garbage that's not captured in your config-on-disk? Better figure it out and get it into env vars, and have your startup script use those to transparently manage whatever bad decisions you made re: state in the past. Shit files all over the system? Better get that sorted out so people can handle persistence with at the very most three total mounts—and oh, gee, look, now your simple example docker-compose also serves to document where exactly you store files. And so on.

(my second-favorite thing is that it's a de-facto cross-distro package manager with very up-to-date packages that are trivial to completely and cleanly uninstall)

Re: Execute Docker Containers as QEMU MicroVMs

#32

Post by [deleted] »

riobard wrote: post #5

A few years ago I invested in a small startup called `hyper.sh`. It open sourced a container runtime called `runV` which provided exactly this: security of virtual machines plus convenience of containers. The project later merged with Intel Clear Container to become what's now called Kata Containers ( https://katacontainers.io/ ) and is now widely used by several Internet giants like Alibaba and Baidu. The startup wa…

[deleted]

Re: Execute Docker Containers as QEMU MicroVMs

#33

Post by nicolaslem »

handrous wrote: post #12
encryptluks2 wrote: post #8

Why not run containers in VMs in containers in VMs? :) Seriously, VMs are hardly as secure as many people want to believe unless you're utilizing enclaves and even that has vulnerabilities. I think a better approach is Seccomp and whatever other filtering makes sense.

A while back I did some looking at FreeBSD jails to try to figure out why they don't have more mindshare (especially when paired with the nigh-superpower-granting ZFS). I came away baffled that they weren't more widely-promoted, compared with Docker and friends. After thinking about it for a while, all I can figure is they're so straightforward to use and well-documented that there's no room to make one's name, or to…

TrueNAS exposed me to FreeBSD jails but what put me off is that there does not seem to be an equivalent of "docker build".

Jails seem to be treated like OpenVZ containers in the Linux world: a lighter alternative to virtual machines, not a way to build and distribute applications like Docker.

This is just my take after playing a few hours with jails, I would happily be proven wrong.

Re: Execute Docker Containers as QEMU MicroVMs

#35

Post by tptacek »

encryptluks2 wrote: post #8

Why not run containers in VMs in containers in VMs? :) Seriously, VMs are hardly as secure as many people want to believe unless you're utilizing enclaves and even that has vulnerabilities. I think a better approach is Seccomp and whatever other filtering makes sense.

I don't know what people generally believe.

But the attack surface of a Linux kernel is very large, is pretty unpredictable, and can't be coherently masked out with rules (my favorite example Jann Horn's VM reference count bug, which was a simple concurrency flaw in the core virtual memory system). By comparison, a Linux KVM hypervisor is not just a subset of the kernel by definition, but also a much smaller codebase, a tiny fraction of the whole kernel.

Replacing shared-kernel isolation like seccomp-filtered containers with VMs is, architecturally, simply the replacement of a large trusted computing base with a smaller one. If the overhead is acceptable, it's hard to argue with from a security perspective.

Re: Execute Docker Containers as QEMU MicroVMs

#36

Post by tptacek »

ashishbijlani wrote: post #24
mikepurvis wrote: post #22

Earlier quoted context omitted.

Is this what gvisor is? https://github.com/google/gvisor

No, gVisor is from Google. They emulate system calls in user-space and use VMs, which increases runtime performance overhead. We use hardware virtualization to directly run containers -- no I/O emulation, no expensive VM exits, scale as needed. Initial comparison with FC/GVisor/Xen here: https://github.com/ashishbijlani/kwarantine

It sounds like you just said "yes, but what we're building is faster". The userland Linux emulation is a security benefit, not a liability.

Re: Execute Docker Containers as QEMU MicroVMs

#37

Post by tptacek »

handrous wrote: post #12
encryptluks2 wrote: post #8

Why not run containers in VMs in containers in VMs? :) Seriously, VMs are hardly as secure as many people want to believe unless you're utilizing enclaves and even that has vulnerabilities. I think a better approach is Seccomp and whatever other filtering makes sense.

A while back I did some looking at FreeBSD jails to try to figure out why they don't have more mindshare (especially when paired with the nigh-superpower-granting ZFS). I came away baffled that they weren't more widely-promoted, compared with Docker and friends. After thinking about it for a while, all I can figure is they're so straightforward to use and well-documented that there's no room to make one's name, or to…

Jails are still shared-kernel isolation. Docker's reputation is mired in its earlier implementations, when it wasn't really even intended for multitenant isolation. Modern Docker, running with unprivileged containers (which is the norm), is substantially hardened. The real win over Docker is losing the shared kernel, which is what lots of people are doing, so the win to Jails is marginal.

Re: Execute Docker Containers as QEMU MicroVMs

#38

Post by tptacek »

riobard wrote: post #9
encryptluks2 wrote: post #8

Why not run containers in VMs in containers in VMs? :) Seriously, VMs are hardly as secure as many people want to believe unless you're utilizing enclaves and even that has vulnerabilities. I think a better approach is Seccomp and whatever other filtering makes sense.

That's the approach taken by Google's gVisor (at the cost of I/O and network performance).

No, that's really not at all what gVisor is. gVisor is best thought of as user-mode Linux --- a complete reimplementation of most of the OS kernel. It's not a system call filter; it's something much closer to a VM than to seccomp.

gVisor is a very cool codebase. As an illustration of the approach: it includes its own TCP/IP stack; we use it in our command-line dev tool to allow people to SSH to their VMs over WireGuard without having to install WireGuard or obtain privileges to manage WireGuard.

Re: Execute Docker Containers as QEMU MicroVMs

#39

Post by tptacek »

As I understand the landscape here, the big enabling win of microvms is faster boot time; there's a cool qemu-lite slide deck that goes into detail about how they cut down boot time:

https://www.linux-kvm.org/images/d/d2/03x05B-Chao_Peng-Light...

The big win was slashing away the BIOS stuff.

We use AWS's Firecracker to turn our customers Docker containers into Firecracker microvms (Firecracker is Amazon's Rust VMM, the engine for Fargate and Lambda). Anecdotally: in my dev environment, the difference between Firecracker boot times and native Docker container startup is imperceptible; the logging we do swamps the VM boot stuff. It's very fast.

Re: Execute Docker Containers as QEMU MicroVMs

#40

Post by lifty »

riobard wrote: post #5

A few years ago I invested in a small startup called `hyper.sh`. It open sourced a container runtime called `runV` which provided exactly this: security of virtual machines plus convenience of containers. The project later merged with Intel Clear Container to become what's now called Kata Containers ( https://katacontainers.io/ ) and is now widely used by several Internet giants like Alibaba and Baidu. The startup wa…

I worked with their tech, testing it, and I loved the product. It was definitely ahead of its time. Similar in some ways to what Fly is doing these days, without the edge.
Post reply on HN