Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced Firecracker with QEMU

hocus.dev

Post reply on HN
Print view View original

81–90 of 156 posts

Re: We replaced Firecracker with QEMU

#81

Post by nerpderp82 »

no_wizard wrote: post #19
gorbak25 wrote: post #6

Earlier quoted context omitted.

I think their usecase makes a lot of sense as their workloads consume a predefined amount of ram. As a customer you rent a VM with a specified amount of memory so fly.io does not care about reclaiming it from a running VM.

Depends on if they're using smart memory allocation to keep costs lower, IE, if they can pattern that certain workloads only need N amount of memory at Y time, they can effectively borrow memory from one VM for usage in another that has an opposite statistical likelihood of needing that memory. This is why paying for dedicated memory is often more expensive than its counter part, because that dedicated memory is not…

Shit like this is how you burn customers and then they find out you aren't changing the sheets, they change hotels.

Just because you can doesn't mean you should.

Re: We replaced Firecracker with QEMU

#82

Post by bonzini »

foundry27 wrote: post #56
gwd wrote: post #3

Listen people, Firecracker is NOT A HYPERVISOR. A hypervisor runs right on the hardware. KVM is a hypervisor. Firecracker is a process that controls KVM. If you want to call firecracker (and QEMU, when used in conjunction with KVM) a VMM ("virtual machine monitor") I won't complain. But please please please, we need a word for what KVM and Xen are, and "hypervisor" is the best fit. Stop using that word for a user-lev…

Nitpick: it’s not accurate to say that a hypervisor, by definition, runs right on the hardware. Xen (as a type-1 hypervisor) has this property; KVM (as a type-2 hypervisor) does not. It’s important to remember that the single core responsibility of a hypervisor is to divide hardware resources and time between VMs, and this decision-making doesn’t require bare-metal. For those unfamiliar, the informal distinction betw…

According to the actual paper that introduced the distinction, and adjusting for change of terminology in the last 50 years, a type-1 hypervisor runs in kernel space and a type-2 hypervisor runs in user space. x86 is not virtualizable by a type-2 hypervisor, except by software emulation of the processor.

What actually can change is the amount of work that the kernel-mode hypervisor leaves to a less privileged (user space) component.

For more detail see https://www.spinics.net/lists/kvm/msg150882.html

Re: We replaced Firecracker with QEMU

#83

Post by bonzini »

orangepurple wrote: post #75
bonzini wrote: post #51

Earlier quoted context omitted.

Pretty good I must say. AWS however is using KVM only these days!

What is AWS Nitro then?

Nitro is the custom hardware that runs in AWS data centers (or in outposts). You can use it bare metal, or if you use VM instances it will run Linux (plus KVM) and a custom user space component that replaces QEMU.

Re: We replaced Firecracker with QEMU

#84

Post by monocasa »

veber-alex wrote: post #76
foundry27 wrote: post #56

Earlier quoted context omitted.

Nitpick: it’s not accurate to say that a hypervisor, by definition, runs right on the hardware. Xen (as a type-1 hypervisor) has this property; KVM (as a type-2 hypervisor) does not. It’s important to remember that the single core responsibility of a hypervisor is to divide hardware resources and time between VMs, and this decision-making doesn’t require bare-metal. For those unfamiliar, the informal distinction betw…

KVM is a type-1 hypervisor [1] [1]: https://www.redhat.com/en/topics/virtualization/what-is-KVM

There's arguments in both directions for something like kvm. Wiki states it pretty well:

> The distinction between these two types is not always clear. For instance, KVM and bhyve are kernel modules[6] that effectively convert the host operating system to a type-1 hypervisor.[7] At the same time, since Linux distributions and FreeBSD are still general-purpose operating systems, with applications competing with each other for VM resources, KVM and bhyve can also be categorized as type-2 hypervisors.[8]

https://en.wikipedia.org/wiki/Hypervisor#Classification

Re: We replaced Firecracker with QEMU

#85

Post by bkettle »

yjftsjthsd-h wrote: post #37
sheepscreek wrote: post #30

Earlier quoted context omitted.

..so is it more to support directly deploying functions to the cloud? Like, what AWS Lambda and CloudFront Functions might be built on?

I'm pretty sure firecracker was literally created to underlie AWS Lambda. EDIT: Okay, https://www.geekwire.com/2018/firecracker-amazon-web-service... says my "pretty sure" memory is in fact correct.

As does the paper [1] with details in section 4.1.

[1]: https://www.usenix.org/system/files/nsdi20-paper-agache.pdf

Re: We replaced Firecracker with QEMU

#86

Post by solarpunk »

doliveira wrote: post #67
jacobr1 wrote: post #43

Earlier quoted context omitted.

[flagged]

Honestly, when we're discussing with humans we want the humans' response. Anyone of us can ask ChatGPT, I don't know why people keep pasting its answers

people keep doing it because it lets them farm upvotes.

Re: We replaced Firecracker with QEMU

#87

Post by CompuIves »

rgbrenner wrote: post #28

"Firecracker's RAM footprint starts low, but once a workload inside allocates RAM, Firecracker will never return it to the host system." Firecracker has a balloon device you can inflate (ie: acquire as much memory inside the VM as possible) and then deflate... returning the memory to the host. You can do this while the VM is running. https://github.com/firecracker-microvm/firecracker/blob/main...

Yes, we use this at CodeSandbox for reclaiming memory to the host (and to reduce snapshot size when we hibernate the VM).

Re: We replaced Firecracker with QEMU

#88

Post by monocasa »

theossuary wrote: post #44
arun-mani-j wrote: post #23

Earlier quoted context omitted.

I think you could help me answer the question that has been in my mind for a month :) Is there any article that tells the difference and relationship between KVM, QEMU, libvirt, virt-manager, Xen, Proxmox etc. with their typical use cases?

KVM is a Linux kernel implementation of the cpu extensions to accelerate vms to near bare metal speeds. Qemu is a user space system emulator. It can emulate in software different architectures like ARM, x86, etc. It can also emulate drivers, networking, disks, etc. Is called via the command line. The reason you'll see Qemu/KVM a lot is because Qemu is the emulator, the things actually running the VM. And it utilizes…

> I haven't played much with xen though, KVM is kind of the defacto, but iirc AWS uses a modified Xen.

AWS switched to KVM, and even a lot of AWS systems that report themselves as Xen are running as KVM with a compat shim to say it's Xen.

Re: We replaced Firecracker with QEMU

#89

Post by CompuIves »

At CodeSandbox we use Firecracker for hosting development environments, and I agree with the points. Though I don't think that means you should not use Firecracker for running long-lived workloads.

We reclaim memory with a memory balloon device, for the disk trimming we discard (& compress) the disk, and for i/o speed we use io_uring (which we only use for scratch disks, the project disks are network disks).

It's a tradeoff. It's more work and does require custom implementations. For us that made sense, because in return we get a lightweight VMM that we can more easily extend with functionality like memory snapshotting and live VM cloning [1][2].

[1]: https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s...

[2]: https://codesandbox.io/blog/cloning-microvms-using-userfault...

Re: We replaced Firecracker with QEMU

#90

Post by [deleted] »

gwd wrote: post #3

Listen people, Firecracker is NOT A HYPERVISOR. A hypervisor runs right on the hardware. KVM is a hypervisor. Firecracker is a process that controls KVM. If you want to call firecracker (and QEMU, when used in conjunction with KVM) a VMM ("virtual machine monitor") I won't complain. But please please please, we need a word for what KVM and Xen are, and "hypervisor" is the best fit. Stop using that word for a user-lev…

[deleted]
Post reply on HN