We replaced Firecracker with QEMU

Fly.io themselves admitted they’re oversubscribed and AWS is doing the same for years now

Source: https://fly.io/blog/the-serverless-server/

Light is cool but for many tasks that level of Spartan is overkill

If I’m investing time in light it might as well be wasm tech

Maybe it's because of the time I grew up in, but in my mind the prototypical Type-I hypervisor is VMWare ESX Server; and the prototypical Type-II hypervisor is VMWare Workstation.

It should be noted that VMWare Workstation always required a kernel module (either on Windows or Linux) to run; so the core "hypervisor-y" bit runs in kernel mode either way. So what's the difference?

The key difference between those two, to me is: Is the thing at the bottom designed exclusively to run VMs, such that every other factor gives way? Or does the thing at the bottom have to "play nice" with random other processes?

The scheduler for ESX Server is written explicitly to schedule VMs. The scheduler for Workstation is the Windows scheduler. Under ESX, your VMs are the star of the show; under Workstation, your VMs are competing with the random updater from the printer driver.

Xen is like ESX Sever: VMs are the star of the show. KVM is like Workstation: VMs are "just" processes, and are competing with whatever random bash script was created at startup.

KVM gets loads of benefits from being in Linux; like, it had hypervisor swap from day one, and as soon as anyone implements something new (like say, NUMA balancing) for Linux, KVM gets it "for free". But it's not really for free, because the cost is that KVM has to make accommodations to all the other use cases out there.

> There's no performance benefit to type-1 - a lot of performance sits on the device emulation side, and both are going to default to qemu there.

Er, both KVM and Xen try to switch to paravirtualized interfaces as fast as possible, to minimize the emulation that QEMU has to do.

And then they learn all hotels are doing exactly same thing. One hotel doing is risk, all hotels doing is industry standard.

Airlines, hotels, restaurants, doctors and so on oversubscribe all the time. Whoever complains are free to move and add to their further disappointments.

My point is that these are largely appropriated terms - neither would fit the definitions of type 1 or type 2 from the early days when Popek and Goldberg were writing about them.

> Or does the thing at the bottom have to "play nice" with random other processes?

From this perspective, Xen doesn't count. You can have all sorts of issues from the dom0 side and competing with resources - you mention PV drivers later, and you can 100% run into issues with VMs because of how dom0 schedules blkback and netback when competing with other processes.

ESXi can also run plenty of unmodified linux binaries - go back in time 15 years and it's basically a fully featured OS. There's a lot running on it, too. Meanwhile, you can build a linux kernel with plenty of things switched off and a root filesystem with just the bare essentials for managing kvm and qemu that is even less useful for general purpose computing than esxi.

>Er, both KVM and Xen try to switch to paravirtualized interfaces as fast as possible, to minimize the emulation that QEMU has to do.

There are more things being emulated than there are PV drivers for, but this is a bit outside of my point.

For KVM, the vast majority of implementations are using qemu for managing their VirtIO devices as well - https://developer.ibm.com/articles/l-virtio/ - you'll notice that IBM even discusses these paravirtual drivers directly in context of "emulating" the device. Perhaps a better way to get the intent across here would be saying qemu handles the device model.

From a performance perspective, ideally you'd want to avoid PV here too and go with sr-iov devices or passthrough.