Live data from Hacker News

We replaced Firecracker with QEMU

hocus.dev

111–120 of 156 posts

Re: We replaced Firecracker with QEMU

#111

I really want VM's to integrate 'smarter' with the host. For example, if I'm running 5 VM's, there is a good chance that many of the pages are identical. Not only do I want those pages to be deduplicated, but I want them to be zero-copy (ie. not deduplicated after-the-fact by some daemon). To do that, the guest block cache needs to be integrated with the host block-cache, so that whenever some guest application tries…

If you already know so much about your application(s), are you sure you need virtualization?

The second I read "shared block cache" my brain went to containers.

If you want data colocated on the same filesystem, then put it on the same filesystem. VMs suck, nobody spins up a whole fabricated IBM-compatible PC and gaslights their executable because they want to.[1] They do it because their OS (a) doesn't have containers, (b) doesn't provide strong enough isolation between containers, or (c) the host kernel can't run their workload. (Different ISA, different syscalls, different executable format, etc.)

Anyone who has ever tried to run heavyweight VMs atop a snapshotting volume already knows the idea of "shared blocks" is a fantasy; as soon as you do one large update inside the guest the delta between your volume clones and the base snapshot grows immensely. That's why Docker et al. has a concept of layers and you describe your desired state as a series of idempotent instructions applied to those layers. That's possible because Docker operates semantically on a filesystem; much harder to do at the level of a block device.

Is the a block containing b"hello, world" part of a program's text section, or part of a user's document? You don't know, because the guest is asking you for an LBA, not a path, not modes, not an ACL, etc. - If you don't know that, the host kernel has no idea how the page should be mapped into memory. Furthermore storing the information to dedup common blocks is non-trivial: go look at the manpage for ZFS' deduplication and it is littered w/ warnings about the performance, memory, and storage implications of dealing with the dedup table.

[1]: https://www.youtube.com/watch?v=coFIEH3vXPw

Re: We replaced Firecracker with QEMU

#112
post #99

Earlier quoted context omitted.

Semantics, libvirt abstracts over KVM via QEMU because QEMU/KVM/HVT is all one driver.

KVM is not enough to create a virtual machine. KVM only virtualizes the processor, not a whole machine.

>Using KVM, one can run multiple virtual machines running unmodified Linux or Windows images. Each virtual machine has private virtualized hardware: a network card, disk, graphics adapter, etc.

Straight from their site. QEMU is the user space interface, KVM the kernel space driver. It’s enough to run whatever OS. That’s the point.

For libvirt: https://libvirt.org/drivers.html

They support a bunch as well.

Re: We replaced Firecracker with QEMU

#113

Earlier quoted context omitted.

Shit like this is how you burn customers and then they find out you aren't changing the sheets, they change hotels. Just because you can doesn't mean you should.

Super common though. Everyone from Hetzner to AWS does this, unless you specifically pay for full allocation, which is typically ~2x the price. Chances are you are using systems that do this and you haven't even noticed.

Werner Vogels seems to disagree: https://twitter.com/Werner/status/25137574680

Re: We replaced Firecracker with QEMU

#114
post #16

I really want VM's to integrate 'smarter' with the host. For example, if I'm running 5 VM's, there is a good chance that many of the pages are identical. Not only do I want those pages to be deduplicated, but I want them to be zero-copy (ie. not deduplicated after-the-fact by some daemon). To do that, the guest block cache needs to be integrated with the host block-cache, so that whenever some guest application tries…

Doubt it is worth the hassle. How many do you really expect to be identical? An OS isn't large. Your spotify/slack/browser instance is of comparable size. Says more about browser based apps but still.

> An OS isn't large. Your spotify/slack/browser instance is of comparable size.

A fairly recent Windows 11 Pro image is ~26GB unpacked and 141k dirents. After finishing OOBE it's already running like >100 processes, >1000 threads, and >100k handles. My Chrome install is ~600MB and 115 dirents. (Not including UserData.) It runs ~1 process per tab. Comparable in scope and complexity? That's debatable, but I tend to agree that modern browsers are pretty similar in scope to what an OS should be. (The other day my "web browser" flashed the firmware on the microcontroller for my keyboard.)

They're not even close to "being comparable in size," although I guess that says more about Windows.

Re: We replaced Firecracker with QEMU

#115
post #99

Earlier quoted context omitted.

KVM is not enough to create a virtual machine. KVM only virtualizes the processor, not a whole machine.

>Using KVM, one can run multiple virtual machines running unmodified Linux or Windows images. Each virtual machine has private virtualized hardware: a network card, disk, graphics adapter, etc. Straight from their site. QEMU is the user space interface, KVM the kernel space driver. It’s enough to run whatever OS. That’s the point. For libvirt: https://libvirt.org/drivers.html They support a bunch as well.

Exactly, it's QEMU that abstracts over the processor virtualization APIs of KVM, Microsoft WHPX, Apple Hypervisor.framework etc. Not Libvirt.

Re: We replaced Firecracker with QEMU

#116
Tl;dr: We tried to misuse technology and we failed. If Firecracker was developed for a single binary executed fir a short period of time why do you try to use it for multiple executables running for a long time? Does it make any sense to even try?

Re: We replaced Firecracker with QEMU

#117
post #45

Earlier quoted context omitted.

Ps: typically if you want to run VMs you are faced with two paths only: 1) you want one or more VMs on your personal computer or 2) you want one or more VMs in an office environment. On the first path you are likely going to be just fine with VirtualBox, VMWare Workstation or Hyper-V (Windows only) / Parallels (Mac intended). Which one you should pick depends on your desired use of the machines. On the second path yo…

> On the first path you are likely going to be just fine with VirtualBox, VMWare Workstation or Hyper-V (Windows only) / Parallels (Mac intended). As a Linux user, why would you want to use VirtualBox or VMWare Workstation? They are not so well integrated with the system, and, frankly, VirtualBox is more of a toy VM player... just go for virt-manager. It gives a conceptually similar interface to VirtualBox, but bette…

The only reason for using it would be that the GUI is a little bit more capable. But feature crippling without the PUEL addons is horrible.

Re: We replaced Firecracker with QEMU

#118
post #114
post #16

Earlier quoted context omitted.

Doubt it is worth the hassle. How many do you really expect to be identical? An OS isn't large. Your spotify/slack/browser instance is of comparable size. Says more about browser based apps but still.

> An OS isn't large. Your spotify/slack/browser instance is of comparable size. A fairly recent Windows 11 Pro image is ~26GB unpacked and 141k dirents. After finishing OOBE it's already running like >100 processes, >1000 threads, and >100k handles. My Chrome install is ~600MB and 115 dirents. (Not including UserData.) It runs ~1 process per tab. Comparable in scope and complexity? That's debatable, but I tend to agr…

My reading was that the "comparable in size" was more about memory footprint and less about usage of storage

Re: We replaced Firecracker with QEMU

#119
post #99

Earlier quoted context omitted.

KVM is not enough to create a virtual machine. KVM only virtualizes the processor, not a whole machine.

>Using KVM, one can run multiple virtual machines running unmodified Linux or Windows images. Each virtual machine has private virtualized hardware: a network card, disk, graphics adapter, etc. Straight from their site. QEMU is the user space interface, KVM the kernel space driver. It’s enough to run whatever OS. That’s the point. For libvirt: https://libvirt.org/drivers.html They support a bunch as well.

I don't want to necessarily make this an argument to/from authority, but for some context here - you are discussing this with Paolo Bonzini, maintainer of KVM, contributor to QEMU. In the list of people that best understand the difference and demarcation points between KVM and QEMU, he's pretty far up there.

Re: We replaced Firecracker with QEMU

#120

Tl;dr: We tried to misuse technology and we failed. If Firecracker was developed for a single binary executed fir a short period of time why do you try to use it for multiple executables running for a long time? Does it make any sense to even try?

Where in the "sales" pitch on the fancy-CSS website as well as the README does it say only to use it for single-shot workloads?

I think the complaints are perfectly valid.

Post reply on HN