What was the hardest part/most unexpected design challenge in getting this to work?
Scaling to 1M concurrent sandboxes in seconds
11–16 of 16 posts
Re: Scaling to 1M concurrent sandboxes in seconds
#12Every scheduler node has cached view of whole cluster and optimistically makes a scheduling decision, retrying on conflict? Any tricks you did to reduce conflict rate? Is there a certain cluster saturation threshold (little free capacity) where conflict rates would get too high?
We try to spread out sandboxes evenly across the cluster (at least, across the workers which are available to take new sandboxes) to minimize conflict. But in general we don't get close to saturation thresholds so high that conflict becomes a problem, except during massive load tests. I suspect we'd see issues around 90% effective utilization.
Makes sense thanks!
Re: Scaling to 1M concurrent sandboxes in seconds
#13> While we do need to write sandbox metadata and results to durable storage, we do so largely asynchronously.
How do you guarantee durability of sandbox task metadata if it’s written to durable storage async? What if the node it’s scheduled on goes down right after scheduling completes - what service durably knows about the intended state of the sandbox and retries scheduling?
Re: Scaling to 1M concurrent sandboxes in seconds
#14I'm a huge scheduling nerd, and the container scheduling system in this post is probably the most impactful system I've worked on. It's quite different than existing solutions, and I personally feel it's at an interesting point in the design space -- very distributed, no strong consistency anywhere, and oriented towards massive scales. Would love to hear feedback and thoughts!
Definitely interesting to read about. It seems like it has some similarities to the approach Fly.io blogged about here: https://fly.io/blog/carving-the-scheduler-out-of-our-orchest...
Re: Scaling to 1M concurrent sandboxes in seconds
#15You can scale firecracker vms like really fast. In our shop we have a simple go service that maintain the state in a sqlite database. The only requirement is bare-metal servers.
how complex is maintaining a firecracker cluster over bare metal in practice ?
Re: Scaling to 1M concurrent sandboxes in seconds
#16You can scale firecracker vms like really fast. In our shop we have a simple go service that maintain the state in a sqlite database. The only requirement is bare-metal servers.
how complex is maintaining a firecracker cluster over bare metal in practice ?
We don't know what we don't know but it does not look like that complex. We have not hit any edge-cases yet. I am sure there are some.