I wonder if providers like Hetzner and Digital Ocean etc. will get this someday also.
AWS Adds support for nested virtualization
31–40 of 124 posts
Re: AWS Adds support for nested virtualization
#32Earlier quoted context omitted.
I was an Amazon EC2 Specialist SA in a prior role, so I know a little about this. If EC2 were like your home server, you might be right. And an EC2 bare metal instance is the closest approximation to that. On bare metal, you've always been free to run your own VMs, and we had some customers who rolled their own nested VM implementations on it. But EC2 is not like your home server. There are some nontrivial considerat…
Seriously curious, don’t Firecracker VMs already run on EC2 instances under the hood when they host Lambda and Fargate?
Unfortunately I'm not at liberty to dive deep into those details. I will say that Firecracker can be used on bare metal EC2 instances, whether you're a public customer or AWS itself. :-)
Re: AWS Adds support for nested virtualization
#33Earlier quoted context omitted.
I was an Amazon EC2 Specialist SA in a prior role, so I know a little about this. If EC2 were like your home server, you might be right. And an EC2 bare metal instance is the closest approximation to that. On bare metal, you've always been free to run your own VMs, and we had some customers who rolled their own nested VM implementations on it. But EC2 is not like your home server. There are some nontrivial considerat…
Seriously curious, don’t Firecracker VMs already run on EC2 instances under the hood when they host Lambda and Fargate?
Since I don't work for AWS I'm allowed to say that at the scale of millions/billions of microVMs you're better off running them on bare metal instances to avoid the overhead of nested virtualization.
Re: AWS Adds support for nested virtualization
#34I wonder if this is connected to Azure launching OpenShift Virtualization on "Boost" SKUs? There are a lot of VMWare customers going to OpenShift Virt, and apparently the CPU/memory overhead on Azure maxes out around 10% under full load... but then hyper V has been doing a lot of work on it. No idea if nitro includes any of the KVM-on-KVM passthrough of full KVM, to give it an edge here.
Re: AWS Adds support for nested virtualization
#35Earlier quoted context omitted.
Seriously curious, don’t Firecracker VMs already run on EC2 instances under the hood when they host Lambda and Fargate?
Unfortunately I'm not at liberty to dive deep into those details. I will say that Firecracker can be used on bare metal EC2 instances, whether you're a public customer or AWS itself. :-)
I guess I should have peeked at the source code when I was there…
Re: AWS Adds support for nested virtualization
#36I wonder if this will extend SEV-SNP and TDX to the child VMs?
Re: AWS Adds support for nested virtualization
#37> Nested virtualization is supported only on 8th generation Intel-based instance types (c8i, m8i, r8i, and their flex variants). When nested virtualization is enabled, Virtual Secure Mode (VSM) is automatically disabled for the instance.
Re: AWS Adds support for nested virtualization
#38Earlier quoted context omitted.
Yep. It's pretty boring. I've been using it at home for years and years with libvirt on very not-special consumer hardware. I guess the AWS clown is finally catching up on this one little not-new-at-all thing.
I was an Amazon EC2 Specialist SA in a prior role, so I know a little about this. If EC2 were like your home server, you might be right. And an EC2 bare metal instance is the closest approximation to that. On bare metal, you've always been free to run your own VMs, and we had some customers who rolled their own nested VM implementations on it. But EC2 is not like your home server. There are some nontrivial considerat…
There's no better way to get good information that is right, than to say something that is misguided and/or wrong.
Thanks for the well-reasoned response.
Re: AWS Adds support for nested virtualization
#39I wonder if this will extend SEV-SNP and TDX to the child VMs?
It says VSM is automatically disabled... so I would assume not.
Re: AWS Adds support for nested virtualization
#40Is nested VMX virtualization in the Linux kernel really that stable?
The technical details are a lot more complex than most realize.
Single level VMX virtualization is relatively straightforward even if there are a lot of details to juggle with VMCS setup and handing exits.
Nested virtualization is a whole another animal as one now also has to handle not just the levels but many things the hardware normally does, plus juggling internal state during transitions between levels.
The LKML is filled with discussions and debates where very sharp contributors are trying to make sense of how it would work.
Amazon turning the feature on is one thing. It working 100% perfectly is quite another…