Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced Firecracker with QEMU

hocus.dev

Post reply on HN
Print view View original

61–70 of 156 posts

Re: We replaced Firecracker with QEMU

#61

Post by bonzini »

gabereiser wrote: post #59
bonzini wrote: post #49

Earlier quoted context omitted.

This is almost entirely wrong especially as far as QEMU, Libvirt and virt-manager are concerned. QEMU is a low level process that represents the virtual machine. It has no equivalent in Xen. Using QEMU directly is not a good idea unless your needs for VM configurations change all the time and you hardly reuse VMs. Libvirt is at a higher level than QEMU. It manages the QEMU processes and gives them access to system re…

How? KVM and Xen are kernel level. QEMU uses KVM but also has a software virtualization capability. Libvirt is an API abstraction over it all. virt-manager is a gui app to manage libvirt machines. Proxmox as well. Proxmox VE talks to VMHost via libvirt.

Libvirt does not use KVM. Libvirt uses either QEMU (which in turn might or might not use KVM) or Xen or other hypervisors. So it's incorrect to say that Libvirt abstracts over KVM.

And virt-manager indeed manages Libvirt machines so it's not at the level of QEMU as you wrote in the parent comment:

> Proxmox is a virtual machine manager (like QEMU, virt-manager)

Re: We replaced Firecracker with QEMU

#62

Post by zokier »

yjftsjthsd-h wrote: post #37
sheepscreek wrote: post #30

Earlier quoted context omitted.

..so is it more to support directly deploying functions to the cloud? Like, what AWS Lambda and CloudFront Functions might be built on?

I'm pretty sure firecracker was literally created to underlie AWS Lambda. EDIT: Okay, https://www.geekwire.com/2018/firecracker-amazon-web-service... says my "pretty sure" memory is in fact correct.

That being said, firecracker also runs long-running tasks on AWS in the form of Fargate

Re: We replaced Firecracker with QEMU

#63

Post by adql »

theossuary wrote: post #44
arun-mani-j wrote: post #23

Earlier quoted context omitted.

I think you could help me answer the question that has been in my mind for a month :) Is there any article that tells the difference and relationship between KVM, QEMU, libvirt, virt-manager, Xen, Proxmox etc. with their typical use cases?

KVM is a Linux kernel implementation of the cpu extensions to accelerate vms to near bare metal speeds. Qemu is a user space system emulator. It can emulate in software different architectures like ARM, x86, etc. It can also emulate drivers, networking, disks, etc. Is called via the command line. The reason you'll see Qemu/KVM a lot is because Qemu is the emulator, the things actually running the VM. And it utilizes…

>Xen is an alternative hypervisor (like esxi). Instead of running on top of Linux, Xen has it's own microkernel. This means less flexibility (there's no Linux body running things), but also simpler to manage and less attack surface. I haven't played much with xen though, KVM is kind of the defacto, but iirc AWS uses a modified Xen.

If you actually played with Xen you'd know it's not actually easier to manage. And increased security claims are dubious at best, as same thing that would be attacked (dom0 managing the whole thing and running linux) have direct unfettered access to xen microkernel. There is reason many sites migrated away from Xen to KVM. Also many Xen drivers de facto run part Linux dom0 instance so you don't even get that isolation.

We ran Xen for few years, as KVM at first was still not as refined and Xen was first to mature market, and it was just million little annoying things.

KVM offers far simple and straightforward management. A VM is just a process. You can look at its CPU usage via normal tools. No magic. No driver problems.

Re: We replaced Firecracker with QEMU

#64

Post by adql »

MuffinFlavored wrote: post #20
gwd wrote: post #3

Listen people, Firecracker is NOT A HYPERVISOR. A hypervisor runs right on the hardware. KVM is a hypervisor. Firecracker is a process that controls KVM. If you want to call firecracker (and QEMU, when used in conjunction with KVM) a VMM ("virtual machine monitor") I won't complain. But please please please, we need a word for what KVM and Xen are, and "hypervisor" is the best fit. Stop using that word for a user-lev…

> virtual machine monitor Is it good to think of libvirt as a virtual machine mointor, or is that more "virtual machine management"?

It's a lot of glue to present a consistent interface but it also does the management part.

"API to virtualization system" would probably be closest approximation but it also does some more advanced stuff like coordinating cross-host VM migration

Re: We replaced Firecracker with QEMU

#65

Post by Liquid_Fire »

londons_explore wrote: post #35
re-thc wrote: post #13

Earlier quoted context omitted.

KVM has KSM (kernel samepage merging) since a long time ago that de-duplicates pages.

But that makes a copy first, and only later notices that the pages are the same and merges them again. Better to not make copies in the first place.

How are you going to know in advance that the pages are going to be the same?

e.g. your guest kernel is loading an application into memory, by reading some parts of an ELF file from disk. Presumably each VM has its own unique disk, so the hypervisor can't know that this is "the same" page of data as another VM has without actually reading it into memory first and calculating a hash or something.

If the VMs share a disk image (e.g. the image is copy-on-write), then I could see it being feasible - e.g. with KVM, even if your VMs are instantiated by distinct userspace processes, they would probably share the pages as they mmap the same disk image. You would still need your virtualised disk device to support copy-on-write, which may or may not be possible depending on your use case.

But your copy-on-write disk images will probably quickly diverge in a way that makes most pages not shareable, unless you use some sort of filesystem optimised for that.

Lastly, since you mentioned Chromium or Slack in another comment - I'm sure you'll find nearly all of the loading time there is not spent loading the executable from disk, but actually executing it (and all its startup/initialisation code). So this probably won't be the speedup you're imagining. It would just save memory.

Re: We replaced Firecracker with QEMU

#66

Post by adql »

MuffinFlavored wrote: post #42
zbrozek wrote: post #26

Earlier quoted context omitted.

I'd love to get a clear explanation of what libvirt actually does. As far as I can tell it's a qemu argument assembler and launcher. For my own use-case, I just launch qemu from systemd unit files: https://wiki.archlinux.org/title/QEMU#With_systemd_service

> As far as I can tell it's a qemu argument assembler Raises a few questions to me: Can you use KVM/do KVM stuff without QEMU? Can you do libvirt stuff without QEMU? Hoping the answers to both aren't useless/"technically, but why would you want to?"

> Can you use KVM/do KVM stuff without QEMU?

There are projects doing that althought qemu is the de facto standard and best bet if you don't need to boot your machines in 20ms

> Can you do libvirt stuff without QEMU?

Libvirt have many backends so yes. IIRC it can even manage virtualbox, althought I'm, not sure why anyone would want to

> Hoping the answers to both aren't useless/"technically, but why would you want to?"

...why? Is there a problem kvm+qemu+libvirt doesn't solve for you?

Re: We replaced Firecracker with QEMU

#67

Post by doliveira »

jacobr1 wrote: post #43
arun-mani-j wrote: post #23

Earlier quoted context omitted.

I think you could help me answer the question that has been in my mind for a month :) Is there any article that tells the difference and relationship between KVM, QEMU, libvirt, virt-manager, Xen, Proxmox etc. with their typical use cases?

[flagged]

Honestly, when we're discussing with humans we want the humans' response. Anyone of us can ask ChatGPT, I don't know why people keep pasting its answers

Re: We replaced Firecracker with QEMU

#68

Post by adql »

paulv wrote: post #40
rgbrenner wrote: post #28

"Firecracker's RAM footprint starts low, but once a workload inside allocates RAM, Firecracker will never return it to the host system." Firecracker has a balloon device you can inflate (ie: acquire as much memory inside the VM as possible) and then deflate... returning the memory to the host. You can do this while the VM is running. https://github.com/firecracker-microvm/firecracker/blob/main...

The first footnote says If you squint hard enough, you'll find that Firecracker does support dynamic memory management with a technique called ballooning. However, in practice, it's not usable. To reclaim memory, you need to make sure that the guest OS isn't using it, which, for a general-purpose workload, is nearly impossible

Yeah it's pretty hard problem as you'd need to defragment physical memory (while fixing all the virtual-to-physical mappings) to make contiguous block to free

Re: We replaced Firecracker with QEMU

#69

Post by adql »

no_wizard wrote: post #19
gorbak25 wrote: post #6

Earlier quoted context omitted.

I think their usecase makes a lot of sense as their workloads consume a predefined amount of ram. As a customer you rent a VM with a specified amount of memory so fly.io does not care about reclaiming it from a running VM.

Depends on if they're using smart memory allocation to keep costs lower, IE, if they can pattern that certain workloads only need N amount of memory at Y time, they can effectively borrow memory from one VM for usage in another that has an opposite statistical likelihood of needing that memory. This is why paying for dedicated memory is often more expensive than its counter part, because that dedicated memory is not…

I think it would be easier to just enable KSM (KVM can use it to deduplicate memory and save some RAM on duplicate blocks like people loading same libraries into memory)
Post reply on HN