How is this comparing to Rund? https://www.usenix.org/conference/atc22/presentation/li-ziju...
Light Sleep: Waking VMs in 200ms with eBPF and snapshots
11–20 of 20 posts
Re: Light Sleep: Waking VMs in 200ms with eBPF and snapshots
#12Great post. Not sure if 200ms is fast though, you can definitely boot from zero to pid1 in I guess it depends on the workload, if you are snapshotting an already-loaded Python program, the time savings are huge, but if it's a program with fast startup, it's probably slower to snapshot.
> Waking up instantly on real traffic without breaking clients
is this for new TCP connections? Or also for connections opened prior to sleep?
Re: Light Sleep: Waking VMs in 200ms with eBPF and snapshots
#13Slightly OT but would be cool if there was a way to run computations in some on-demand VM that cold started in 200ms, did it thing, died and you only paid for the time you used it. In essence s lambda that exposed you a full blown VM rather than a limited environment.
There are a few ways to approach this. If you don't mind owning the orchestration layer this is precisely what firecracker does. If you don't even want to pay for that though scheduling unikernels on something like ec2 gets you your full vm, is cheaper, has more resources than lambda and doesn't have the various limitations such as no gpu or timeouts or anything like that.
[deleted]
Re: Light Sleep: Waking VMs in 200ms with eBPF and snapshots
#14Always nice to see folks talking about VM snapshots - they're an extremely powerful tool for building systems of all kinds. At AWS, we use snapshots in Lambda Snapstart (along with cloning, and snapshots are distributed across multiple workers), and in Aurora DSQL (where we clone and restore a snapshot of Postgres on every database connection), in AgentCore Runtime, and a number of other places. > But Firecracker com…
> At AWS, we use snapshots in Lambda Snapstart
I'm curious on why is it taking so long to add support for different runtime? I imagine it would be same for all of them?
> where we clone and restore a snapshot of Postgres on every database connection
This is interesting. Is there any challenge while working on this?
Re: Light Sleep: Waking VMs in 200ms with eBPF and snapshots
#15I feel like I'm missing something here when this is being used with nomad. Caveat being that the only comparable technologies I've worked with are k8s and ECS. In the article they mention that they are using a containerd shim to launch micro VMs, so from the perspective of the scheduler, whether the VM is actually "sleeping" or not, it looks like it's running since they continue to respond to health checks. So what exactly is the point of suspending the VMs on idle if the scheduler still thinks they're running? Whatever memory is reserved for that job is still going to be reserved, so you're not able to oversubscribe the host regardless.
Re: Light Sleep: Waking VMs in 200ms with eBPF and snapshots
#16Always nice to see folks talking about VM snapshots - they're an extremely powerful tool for building systems of all kinds. At AWS, we use snapshots in Lambda Snapstart (along with cloning, and snapshots are distributed across multiple workers), and in Aurora DSQL (where we clone and restore a snapshot of Postgres on every database connection), in AgentCore Runtime, and a number of other places. > But Firecracker com…
> At AWS, we use snapshots in Lambda Snapstart I'm curious on why is it taking so long to add support for different runtime? I imagine it would be same for all of them? > where we clone and restore a snapshot of Postgres on every database connection This is interesting. Is there any challenge while working on this?
From my experience with firecracker, you need to send a signal to the VMM that can be used to indicate the process is "ready" (and the snapshot can be taken).
I assume that every runtime must be forked to add such signal right before calling into usercode
Re: Light Sleep: Waking VMs in 200ms with eBPF and snapshots
#17I feel like I'm missing something here when this is being used with nomad. Caveat being that the only comparable technologies I've worked with are k8s and ECS. In the article they mention that they are using a containerd shim to launch micro VMs, so from the perspective of the scheduler, whether the VM is actually "sleeping" or not, it looks like it's running since they continue to respond to health checks. So what e…
Hey!
You got everything correctly. The advantages are: - For the end-user: not paying or paying less - For the hypervisor owner: a sleeping instance uses no CPU, so it reduces the load on the hypervisor
Other than that, it's still possible to oversubscribe, but you're right, we need to trump the scheduler. Another cool thing is that in the worst case scenario where an hypervisor gets full and it's over capacity, sleeping instances are great candidates for eviction.
Re: Light Sleep: Waking VMs in 200ms with eBPF and snapshots
#18I feel like I'm missing something here when this is being used with nomad. Caveat being that the only comparable technologies I've worked with are k8s and ECS. In the article they mention that they are using a containerd shim to launch micro VMs, so from the perspective of the scheduler, whether the VM is actually "sleeping" or not, it looks like it's running since they continue to respond to health checks. So what e…
Hey! You got everything correctly. The advantages are: - For the end-user: not paying or paying less - For the hypervisor owner: a sleeping instance uses no CPU, so it reduces the load on the hypervisor Other than that, it's still possible to oversubscribe, but you're right, we need to trump the scheduler. Another cool thing is that in the worst case scenario where an hypervisor gets full and it's over capacity, slee…
Ah, I think the part that I didn't consider was that an "idle" VM is not zero CPU cost, unlike a container, so indeed from a hypervisor owner perspective you'd like other active VMs to be able to use that CPU time. But again, doesn't that presuppose oversubscription? If a node is fully reserved, it doesn't matter if all of the running VMs are idle, you're still not going to be able to schedule another job on that node, so your costs remain the same unless you oversubscribe the host and count on the fact that there will be unused capacity available most of the time (similar to AWS Flex instances).
Re: Light Sleep: Waking VMs in 200ms with eBPF and snapshots
#19Earlier quoted context omitted.
Hey! You got everything correctly. The advantages are: - For the end-user: not paying or paying less - For the hypervisor owner: a sleeping instance uses no CPU, so it reduces the load on the hypervisor Other than that, it's still possible to oversubscribe, but you're right, we need to trump the scheduler. Another cool thing is that in the worst case scenario where an hypervisor gets full and it's over capacity, slee…
Ah, I think the part that I didn't consider was that an "idle" VM is not zero CPU cost, unlike a container, so indeed from a hypervisor owner perspective you'd like other active VMs to be able to use that CPU time. But again, doesn't that presuppose oversubscription? If a node is fully reserved, it doesn't matter if all of the running VMs are idle, you're still not going to be able to schedule another job on that nod…
Yes definitely as an operator, you want to oversubscribe hosts. What I was mentioning is that there are still small benefits when an host is not full: the CPU gains _and_, for users, the fact that they're not paying/paying less (even though the operator is still paying for the full underutilized hypervisor, but hey, that's the game)
Re: Light Sleep: Waking VMs in 200ms with eBPF and snapshots
#20Earlier quoted context omitted.
> At AWS, we use snapshots in Lambda Snapstart I'm curious on why is it taking so long to add support for different runtime? I imagine it would be same for all of them? > where we clone and restore a snapshot of Postgres on every database connection This is interesting. Is there any challenge while working on this?
From my experience with firecracker, you need to send a signal to the VMM that can be used to indicate the process is "ready" (and the snapshot can be taken). I assume that every runtime must be forked to add such signal right before calling into usercode
Thank you. This explains it.