Live data from Hacker News

We replaced Firecracker with QEMU

hocus.dev

51–60 of 156 posts

Re: We replaced Firecracker with QEMU

#51
post #43

Earlier quoted context omitted.

I think you could help me answer the question that has been in my mind for a month :) Is there any article that tells the difference and relationship between KVM, QEMU, libvirt, virt-manager, Xen, Proxmox etc. with their typical use cases?

[flagged]

Pretty good I must say. AWS however is using KVM only these days!

Re: We replaced Firecracker with QEMU

#52
post #26

Earlier quoted context omitted.

> virtual machine monitor Is it good to think of libvirt as a virtual machine mointor, or is that more "virtual machine management"?

I'd love to get a clear explanation of what libvirt actually does. As far as I can tell it's a qemu argument assembler and launcher. For my own use-case, I just launch qemu from systemd unit files: https://wiki.archlinux.org/title/QEMU#With_systemd_service

The main important point is that Libvirt takes care of privilege separation.

It makes sure that if your VM and/or QEMU are broken out of, there are extra layers to prevent getting access to the whole physical machine. For example it runs QEMU as a very limited user and, if you're using SELinux, the QEMU process can hardly read any file other than the vm image file.

By contrast the method in the arch wiki runs QEMU as root. QEMU is exposed to all sort of untrusted input, so you really don't want it to run as root.

Libvirt also handles cross machine operations such as live migration, and makes it easier to query a bunch of things from QEMU.

For more info see https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u...

Re: We replaced Firecracker with QEMU

#53
post #22

Earlier quoted context omitted.

OpenVZ does this. If you have 5 VMs each loading the same library then memory is conserved, as I understand it.

kvm does the same with KSM.

Not precisely, in that KSM does it after the fact while OpenVZ has it occur as a consequence of its design, on the loading of the program.

See (OpenVZ) "Containers share dynamic libraries, which greatly saves memory." It's just 1 Linux kernel when you are running OpenVZ containers.

https://docs.openvz.org/openvz_users_guide.webhelp/_openvz_c...

See (KVM/KSM): "KSM enables the kernel to examine two or more already running programs and compare their memory. If any memory regions or pages are identical, KSM reduces multiple identical memory pages to a single page. This page is then marked copy on write."

https://access.redhat.com/documentation/en-us/red_hat_enterp...

In KVM's defense, it supports a much wider range of OSes; OpenVZ only really does different versions of Linux, while KVM can run OpenBSD/FreeBSD/NetBSD/Windows and even OS/2 in addition to Linux.

Re: We replaced Firecracker with QEMU

#54
post #26

Earlier quoted context omitted.

I'd love to get a clear explanation of what libvirt actually does. As far as I can tell it's a qemu argument assembler and launcher. For my own use-case, I just launch qemu from systemd unit files: https://wiki.archlinux.org/title/QEMU#With_systemd_service

> As far as I can tell it's a qemu argument assembler Raises a few questions to me: Can you use KVM/do KVM stuff without QEMU? Can you do libvirt stuff without QEMU? Hoping the answers to both aren't useless/"technically, but why would you want to?"

libvirt supports other hypervisors than QEMU, such as VirtualBox, Xen and LXC.

Re: We replaced Firecracker with QEMU

#55

The article did an ok job of explaining the firecracker limitations they ran into but it was extremely skimpy when it came to qemu and just rushed to the conclusion “we did a lot of work so try our product.”

yeah I was reading so I could find out what they did. I understand that they need to sell their product but jeez. don't leave us hanging like that

I didn't want to go into all the technical details, but we have another write-up that goes into details about RAM management: https://github.com/hocus-dev/hocus/blob/main/rfd/0003-worksp...

Other than making sure we release unused memory to the host, we didn't customize QEMU that much. Although we do have a cool layered storage solution - basically a faster alternative to QCOW2 that's also VMM independent. It's called overlaybd, and was created and implemented in Alibaba. That will probably be another blog post. https://github.com/containerd/overlaybd

Re: We replaced Firecracker with QEMU

#56
post #3

Listen people, Firecracker is NOT A HYPERVISOR. A hypervisor runs right on the hardware. KVM is a hypervisor. Firecracker is a process that controls KVM. If you want to call firecracker (and QEMU, when used in conjunction with KVM) a VMM ("virtual machine monitor") I won't complain. But please please please, we need a word for what KVM and Xen are, and "hypervisor" is the best fit. Stop using that word for a user-lev…

Nitpick: it’s not accurate to say that a hypervisor, by definition, runs right on the hardware. Xen (as a type-1 hypervisor) has this property; KVM (as a type-2 hypervisor) does not. It’s important to remember that the single core responsibility of a hypervisor is to divide hardware resources and time between VMs, and this decision-making doesn’t require bare-metal.

For those unfamiliar, the informal distinction between type-1 and type-2 is that type-1 hypervisors are in direct control of the allocation of all resources of the physical computer, while type-2 hypervisors operate as some combination of being “part of” / “running on” a host operating system, which owns and allocates the resources. KVM (for example) gives privileged directions to the Linux kernel and its virtualization kernel module for how to manage VMs, and the kernel then schedules and allocates the appropriate system resources. Yes, the type-2 hypervisor needs kernel-mode primitives for managing VMs, and the kernel runs right on the hardware, but those primitives aren’t making management decisions for the division of hardware resources and time between VMs. The type-2 hypervisor is making those decisions, and the hypervisor is scheduled by the OS like any other user-mode process.

Re: We replaced Firecracker with QEMU

#57
post #3

Listen people, Firecracker is NOT A HYPERVISOR. A hypervisor runs right on the hardware. KVM is a hypervisor. Firecracker is a process that controls KVM. If you want to call firecracker (and QEMU, when used in conjunction with KVM) a VMM ("virtual machine monitor") I won't complain. But please please please, we need a word for what KVM and Xen are, and "hypervisor" is the best fit. Stop using that word for a user-lev…

I think people just pick the coolest sounding term. Imagine someone is sharing what they are working on, what’s cooler sounding “I am working on a virtual machine monitor” or “I am working on a hypervisor”. Hypervisor just sounds futuristic and awesome.

It’s like with “isomorphic” code. That just sounds much cooler than “js that runs on the client and the server”.

Re: We replaced Firecracker with QEMU

#58
post #54

Earlier quoted context omitted.

> As far as I can tell it's a qemu argument assembler Raises a few questions to me: Can you use KVM/do KVM stuff without QEMU? Can you do libvirt stuff without QEMU? Hoping the answers to both aren't useless/"technically, but why would you want to?"

libvirt supports other hypervisors than QEMU, such as VirtualBox, Xen and LXC.

I know ChatGPT responses get downvoted here but I fed it this to help me better understand this:

> is KVM a hypervisor? is it type 1 or type 2? is QEMU a hypervisor, is it type 1 or type 2? if QEMU is using KVM, is QEMU then not a hypervisor in that use case?

Yes, KVM (Kernel-Based Virtual Machine) is indeed a hypervisor. It's a type 1 hypervisor, also known as a "bare metal" hypervisor. This is because KVM directly runs on the host's hardware to control the hardware and to manage guest operating systems. The fact that it's a Linux kernel module that allows the Linux kernel to function as a hypervisor makes it very efficient.

QEMU (Quick Emulator) is a bit more complex. By itself, it is technically a type 2 or "hosted" hypervisor, meaning it runs within a conventional operating system environment. QEMU is a generic, open-source machine emulator and virtualizer that can emulate a variety of hardware types and host a range of guest operating systems.

However, when QEMU is used with KVM, the picture changes somewhat. In this case, KVM provides the hardware virtualization where it allows the host machine to call CPU instructions of full virtualization. Then, QEMU emulates the hardware resources and provides the user interface for the VM, thus allowing for better performance and usability. It's this combination of KVM's hardware acceleration capabilities and QEMU's emulation capabilities that makes them often used together.

In this case, QEMU is not acting purely as a hypervisor; it's providing hardware emulation and user interface for the VMs, while KVM is the part providing the hypervisor functionality. However, we often refer to the combination of "QEMU/KVM" as a unit when talking about this mode of operation.

Re: We replaced Firecracker with QEMU

#59
post #49

Earlier quoted context omitted.

KVM is kernel-based virtual machine, with libvirt being its API abstraction over all of it. QEMU is a virtual machine host that leverages kvm or software virtualization to spin up machines on the host. virt-manager does the same. Xen is another virtual machine host, like KVM. Proxmox is a virtual machine manager (like QEMU, virt-manager) but is web based. Libvirt will provide abstraction for kvm,qemu,xen Use cases: p…

This is almost entirely wrong especially as far as QEMU, Libvirt and virt-manager are concerned. QEMU is a low level process that represents the virtual machine. It has no equivalent in Xen. Using QEMU directly is not a good idea unless your needs for VM configurations change all the time and you hardly reuse VMs. Libvirt is at a higher level than QEMU. It manages the QEMU processes and gives them access to system re…

How? KVM and Xen are kernel level. QEMU uses KVM but also has a software virtualization capability. Libvirt is an API abstraction over it all. virt-manager is a gui app to manage libvirt machines. Proxmox as well. Proxmox VE talks to VMHost via libvirt.

Re: We replaced Firecracker with QEMU

#60
post #26

Earlier quoted context omitted.

I'd love to get a clear explanation of what libvirt actually does. As far as I can tell it's a qemu argument assembler and launcher. For my own use-case, I just launch qemu from systemd unit files: https://wiki.archlinux.org/title/QEMU#With_systemd_service

> As far as I can tell it's a qemu argument assembler Raises a few questions to me: Can you use KVM/do KVM stuff without QEMU? Can you do libvirt stuff without QEMU? Hoping the answers to both aren't useless/"technically, but why would you want to?"

There is cute article from lwn demoing using kvm directly without anything else: https://lwn.net/Articles/658511/
Post reply on HN