Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Sub-millisecond VM sandboxes using CoW memory forking

github.com

Post reply on HN
Print view View original

31–40 of 85 posts

Re: Show HN: Sub-millisecond VM sandboxes using CoW memory forking

#31

Post by adammiribyan »

cperciva wrote: post #22

Don't forget about entropy! You've just created two identical copies of all of your random number generators, which could be very very bad for security. The firecracker team wrote a very good paper about addressing this when they added snapshot support.

Good callout. We seed entropy before snapshot to unblock getrandom(), but forks still share CSPRNG state. The proper fix per Firecracker’s docs is RNDADDENTROPY + RNDRESEEDCRNG after each fork, plus reseeding userspace PRNGs like numpy separately. On the roadmap. https://github.com/firecracker-microvm/firecracker/blob/main...

Re: Show HN: Sub-millisecond VM sandboxes using CoW memory forking

#34

Post by adammiribyan »

diptanu wrote: post #15

The tricky part of doing this in production is cloning sandboxes across nodes. You would have to snapshot the resident memory, file system (or a CoW layer on top of the rootfs), move the data across nodes, etc.

Agreed, cross-node is the hard next step. For now single-node density gets you surprisingly far. 1000 concurrent sandboxes on one $50 box. When we need multi-node, userfaultfd with remote page fetch is the likely path.

Re: Show HN: Sub-millisecond VM sandboxes using CoW memory forking

#35

Post by adammiribyan »

edunteman wrote: post #29
indigodaddy wrote: post #18

Earlier quoted context omitted.

simonw seems like he's always wanting what you describe, maybe more for wasm though

I’ve been a big fan of “what’s the thinnest this could be” interpretations of sandboxes. This is a great example of that. On the other end of the spectrum there’s just-bash from the Vercel folks.

Exactly —- they skip the OS, we make it free to clone.

Re: Show HN: Sub-millisecond VM sandboxes using CoW memory forking

#36

Post by mkj »

adammiribyan wrote: post #31
cperciva wrote: post #22

Don't forget about entropy! You've just created two identical copies of all of your random number generators, which could be very very bad for security. The firecracker team wrote a very good paper about addressing this when they added snapshot support.

Good callout. We seed entropy before snapshot to unblock getrandom(), but forks still share CSPRNG state. The proper fix per Firecracker’s docs is RNDADDENTROPY + RNDRESEEDCRNG after each fork, plus reseeding userspace PRNGs like numpy separately. On the roadmap. https://github.com/firecracker-microvm/firecracker/blob/main...

It looks like firecracker already supports ACPI vmgenid, which will trigger Linux random to reseed? https://github.com/firecracker-microvm/firecracker/blob/main...

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

So that just (!) leaves userspace PRNGs.

Re: Show HN: Sub-millisecond VM sandboxes using CoW memory forking

#38

Post by adammiribyan »

wei03288 wrote: post #37

[flagged]

On tail latency: KVM VM creation is 99.5% of the fork cost - create_vm, create_irq_chip, create_vcpu, and restoring CPU state. The CoW mmap is ~4 microseconds regardless of load. P99 at 1000 concurrent is 1.3ms. The mmap CoW page faults during execution are handled transparently by the host kernel and don't contribute to fork latency.

On snapshot staleness: yes, forks inherit all internal state including RNG seeds. For dependency updates you rebuild the template (~15s). No incremental update - full re-snapshot, similar to rebuilding a Docker image.

On the memory number: 265KB is the fork overhead before any code runs. Under real workloads we measured 3.5MB for a trivial print(), ~27MB for numpy operations. But 93% of pages stay shared across forks via CoW. We measured 100 VMs each running numpy sharing 2.4GB of read-only pages with only 1.75MB private per VM. So the real comparison to E2B's ~128MB is more like 3-27MB depending on workload, with most of the runtime memory shared.

Re: Show HN: Sub-millisecond VM sandboxes using CoW memory forking

#39

Post by adammiribyan »

vmg12 wrote: post #11

Does it only work with that specific version of firecracker and only with vms with 1 vcpu? More than the sub ms startup time the 258kb of ram per VM is huge.

1 vCPU per fork currently. Multi-vCPU is doable (per-vCPU state restore in a loop) but would multiply fork time.

On Firecracker version: tested with v1.12, but the vmstate parser auto-detects offsets rather than hardcoding them, so it should work across versions.

Re: Show HN: Sub-millisecond VM sandboxes using CoW memory forking

#40

Post by injidup »

It's so frustrating seeing all this sandbox tooling pop up for linux but windows is soooooo far behind. I mean Windows Sandbox ( https://learn.microsoft.com/en-us/windows/security/applicati... ) doesn't even have customizable networking white lists. You can turn networking on or off but that's about as fine grained as it gets. So all of us still having to write desktop windows stuff are left without a good method of easily putting our agents in a blast proof box.
Post reply on HN