Live data from Hacker News

We replaced Firecracker with QEMU

hocus.dev

131–140 of 156 posts

Re: We replaced Firecracker with QEMU

#131
post #113

Earlier quoted context omitted.

Werner Vogels seems to disagree: https://twitter.com/Werner/status/25137574680

Tweeted Sep 21, 2010, might still be true for EC2 VMs though.

There's the 't' series of instances that offer burstable CPU. AFAIK still 1:1 on memory though, and there's models that allow you to pay to stay un-throttled when using t family instances vs. getting throttled when out of burst credits.

Re: We replaced Firecracker with QEMU

#132
I came to the same conclusion as OP. QEMU is the most stable, hackable, well-supported VM hypervisor on the market. Setting it up is a pain, but once you get it set up with all your custom scripts, you never have to do it again. Ever. Even in your next project.

Re: We replaced Firecracker with QEMU

#133

At CodeSandbox we use Firecracker for hosting development environments, and I agree with the points. Though I don't think that means you should not use Firecracker for running long-lived workloads. We reclaim memory with a memory balloon device, for the disk trimming we discard (& compress) the disk, and for i/o speed we use io_uring (which we only use for scratch disks, the project disks are network disks). It's a t…

These blogs are wonderful. I'd read them before figuring out firecracker snapshot/restore, but wanted to say it here.

> i/o speed we use io_uring

custom io_uring based driver for the VM block devices? or what do you mean here?

Re: We replaced Firecracker with QEMU

#134

Earlier quoted context omitted.

We do, and we'd love to use it in the future. We've found that it's not ready for prime time yet and it's missing some features. The biggest problem was that it does not support discard operations yet. Here's a short writeup we did about VMMs that we considered: https://github.com/hocus-dev/hocus/blob/main/rfd/0002-worksp...

Thanks for the link to the elaboration! FYI footnotes 3 and 4 seem to be swapped.

> footnotes 3 and 4 seem to be swapped

Maybe they are async footnotes and there is a race condition. /s

Re: We replaced Firecracker with QEMU

#136

> The main issue we've had with QEMU is that it has too many options you need to configure. For instance, enabling your VM to return unused RAM to the host requires at least three challenging tasks This just works on Hyper-V Linux guests btw. For all the crap MS gets they do some things very right.

It kind of just works. It’s actually broken under Debian 13 for some reason; memory usage infinitely balloons if the feature is enabled.

13 is debian-testing so I guess Microsoft still has time to make it work - last I checked it wasn't yet on Azure supported list.

Re: We replaced Firecracker with QEMU

#137
post #40

"Firecracker's RAM footprint starts low, but once a workload inside allocates RAM, Firecracker will never return it to the host system." Firecracker has a balloon device you can inflate (ie: acquire as much memory inside the VM as possible) and then deflate... returning the memory to the host. You can do this while the VM is running. https://github.com/firecracker-microvm/firecracker/blob/main...

The first footnote says If you squint hard enough, you'll find that Firecracker does support dynamic memory management with a technique called ballooning. However, in practice, it's not usable. To reclaim memory, you need to make sure that the guest OS isn't using it, which, for a general-purpose workload, is nearly impossible

A bit disingenuous to make a broad sweeping claim, then have a footnote which contradicts that claim, and upon closer inspection even that claim is incorrect.

It's absolutely usable in practice, it just makes oversubscription more challenging.

Re: We replaced Firecracker with QEMU

#138
post #6

Earlier quoted context omitted.

I think their usecase makes a lot of sense as their workloads consume a predefined amount of ram. As a customer you rent a VM with a specified amount of memory so fly.io does not care about reclaiming it from a running VM.

Depends on if they're using smart memory allocation to keep costs lower, IE, if they can pattern that certain workloads only need N amount of memory at Y time, they can effectively borrow memory from one VM for usage in another that has an opposite statistical likelihood of needing that memory. This is why paying for dedicated memory is often more expensive than its counter part, because that dedicated memory is not…

Generalized oversubscription like that is very challenging if not impossible to do securely, since you want to keep workloads isolated to single tenant numa nodes.

E.g. using the firecracker jailer: https://github.com/firecracker-microvm/firecracker/blob/main...

Re: We replaced Firecracker with QEMU

#139

I know a good way to make a process make the most of the hardware and play cooperatively with other processes: don't use virtualization. I will never understand the whole virtual machine and cloud craze. Your operating system is better than any hypervisor at sharing resources efficiently.

In this context (the blog post) and the reason firecracker was created, was to isolate workloads.

And if youre running untrusted code, then using a virtualized environment is the easiest (id even say best) way to go about it.

Post reply on HN