61–70 of 124 posts
This is a big deal because you can now run Firecracker/other microVMs in an AWS VM instead of expensive AWS bare-metal instances. GCP has had nested virtualization for a while.
I wonder if providers like Hetzner and Digital Ocean etc. will get this someday also.
I wonder if this is connected to Azure launching OpenShift Virtualization on "Boost" SKUs? There are a lot of VMWare customers going to OpenShift Virt, and apparently the CPU/memory overhead on Azure maxes out around 10% under full load... but then hyper V has been doing a lot of work on it. No idea if nitro includes any of the KVM-on-KVM passthrough of full KVM, to give it an edge here.
Azure? OpenShift? "I don't think about you at all." — Matt Garman probably
If anyone here can't wait (as it looks like there's very little info on this at the moment..)
I wrote up detailed instructions for Ant Group's KVM-PVM patches. Performance is OK for background servers/tasks, but does take a hit up to 50% on complex builds like Kernels or Go with the K8s client.
DIY/detailed option:
https://blog.alexellis.io/how-to-run-firecracker-without-kvm...
Fully working, pre-built host and guest kernel and rootfs:
https://docs.slicervm.com/tasks/pvm/
I'll definitely be testing this and comparing as soon as it's available. Hopefully it'll be accelerated somewhat compared to the PVM approach. There's still no sign whether those patches will ever end up merged upstream in the Linux Kernel. If you know differently, I'd appreciate a link.
Azure, OCI, DigitalOcean, GCE all support nested virt as an option and do all take a bit of a hit, but it makes for very easy testing / exploration. Bare-metal on Hetzner now has a setup fee of up to 350 EUR.. you can find some stuff with 0 setup fee, but it's usually quite old kit.
Edit: this doesn't look quite as good as the headline.. Options for instances look a bit limited. Someone found some more info here: https://x.com/nanovms/status/2022141660143165598/photo/1
Is nested VMX virtualization in the Linux kernel really that stable? The technical details are a lot more complex than most realize. Single level VMX virtualization is relatively straightforward even if there are a lot of details to juggle with VMCS setup and handing exits. Nested virtualization is a whole another animal as one now also has to handle not just the levels but many things the hardware normally does, plu…
The more interesting signal is that AWS is restricting this to 8th-gen Intel instances only (c8i/m8i/r8i). They're likely leveraging specific microarchitectural improvements in those chips for VMCS shadowing — picking the hardware generation where they can guarantee their reliability bar rather than enabling it broadly and dealing with errata on older silicon. That's actually the careful engineering approach you'd want from a cloud provider.
This is great news for folks that use microVMs - "we only use AWS" has been an issue for our stuff (slicer services/sandboxes/actuated self-hosted GitHub runners) If anyone here can't wait (as it looks like there's very little info on this at the moment..) I wrote up detailed instructions for Ant Group's KVM-PVM patches. Performance is OK for background servers/tasks, but does take a hit up to 50% on complex builds l…
I don't understand what you are paying for here, nested virtualization doesn't need any extra setup for hardware compared to normal one
... or you are saying Hetzner wants 350 EUR for turning on normal virtualization option in BIOS ?
This is great news for folks that use microVMs - "we only use AWS" has been an issue for our stuff (slicer services/sandboxes/actuated self-hosted GitHub runners) If anyone here can't wait (as it looks like there's very little info on this at the moment..) I wrote up detailed instructions for Ant Group's KVM-PVM patches. Performance is OK for background servers/tasks, but does take a hit up to 50% on complex builds l…
> Bare-metal on Hetzner now has a setup fee of up to 350 EUR.. you can find some stuff with 0 setup fee, but it's usually quite old kit. I don't understand what you are paying for here, nested virtualization doesn't need any extra setup for hardware compared to normal one ... or you are saying Hetzner wants 350 EUR for turning on normal virtualization option in BIOS ?
This is a big deal because you can now run Firecracker/other microVMs in an AWS VM instead of expensive AWS bare-metal instances. GCP has had nested virtualization for a while.
You can use an expensive AWS VM instead of an expensive AWS bare–metal image. Does anyone realise how expensive AWS is, even in the best case?
Renting a server from cheaper hosting providers can be massive savings but you now need to re-invent all of the AWS APIs you use or might use and it's big CAPEX time investment. And any new feature you need, whether that's queue, mail gateway or thousand other APIs need to be deployed and managed first before you can even start testing.
It's less work now than it was before just due to amount of tools there are to automate it but it's still more work that you could be spending on improving your product.
Could someone explain why this is might be a big deal? I remember playing with nested virty some years ago and deciding it is a backwards step except for PoC and the like. Given I haven't personally run out of virty gear, I never needed to do a PoC.
There is no real reason to use it on hardware you own; but in case of cloud you just not always have enough to do to excuse paying for whole entire server
Earlier quoted context omitted.
Yep. It's pretty boring. I've been using it at home for years and years with libvirt on very not-special consumer hardware. I guess the AWS clown is finally catching up on this one little not-new-at-all thing.
I was an Amazon EC2 Specialist SA in a prior role, so I know a little about this. If EC2 were like your home server, you might be right. And an EC2 bare metal instance is the closest approximation to that. On bare metal, you've always been free to run your own VMs, and we had some customers who rolled their own nested VM implementations on it. But EC2 is not like your home server. There are some nontrivial considerat…