Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsandbox: Virtual Machines that feel and perform like containers

github.com

Post reply on HN
Print view View original

151–160 of 195 posts

Re: Microsandbox: Virtual Machines that feel and perform like containers

#151

Post by raphinou »

appcypher wrote: post #112
eamann wrote: post #104

> Ever needed to run code you don't fully trust? Then the installation instructions include piping a remote script directly to Bash ... Oh irony ... That said, the concept itself is intriguing.

Your statement initially went over my head. Sorry lol. You can always download the installer script and audit yourself. I will set up proper distribution later.

In case you're interested when you set up proper distribution, I'm working on an open source solution aiming to improve security of downloads from the internet. Our first step is maintaining a mirror of checksums published in GitHub releases at https://github.com/asfaload/checksums/. If you publish a checksums file in your releases it can automatically be mirrored. The checksums mirror is not our end game, but it already protects against changes of released files from the time the mirror was taken. For anyone interested: https://asfaload.com/asfald/

Re: Microsandbox: Virtual Machines that feel and perform like containers

#152

Post by appcypher »

hinkley wrote: post #67

How’s performance? What’s the overhead versus docker? Terraform or Pulumi integration on the horizon?

Wow. Just seeing this. I've not done proper benchmarking yet but rn we are lagging behind in file I/O for the OverlayFS impl

Re: Microsandbox: Virtual Machines that feel and perform like containers

#153

Post by bjackman »

delusional wrote: post #149
bjackman wrote: post #138

Earlier quoted context omitted.

I see your point but even if your VMM is a zillion lines of C++ with emulated devices there are opportunities to secure it that don't exist with a shared-monolithic-kernel container runtime. You can create security boundaries around (and even within!) the VMM. You can make it so an escape into the VMM process has only minimal value, by sandboxing the VMM aggressively. Plus you can absolutely escape the model of C++ e…

Could you elaborate on how you could secure those architectures better? It's unclear to me how being in device firmware or being a VMM provides you with any further abilities. Surely you still have the same fundamental problem of being a shared resource. Intuitively there are differences. The Linux kernel is fucking huge, and anything that could bake the "shared resources" down to less than the entire kernel would be…

When you escape a container generally you can do whatever the kernel can do. There is no further security boundary.

If you escape into a VMM you can do whatever the VMM can do. You can build a system where it can not do very much more than the VM guest itself. By the time the guest boots the process containing the vCPU threads has already lost all its interesting privileges and has no credentials of value.

Similar with device passthrough. It's not very interesting if the device you're passing through ultimately has unchecked access to PCIe but if you have a proper ioMMU set up it should be possible to have a system where pwning the device firmware is just a small step rather than an immediate escalation to root-equivalent. (I should say, I don't know if this system actually exists today, I just know it's possible).

With a VMM escape your next step is usually to exploit the kernel. But if you sandbox the VMM properly there is very limited kernel attack surface available to it.

So yeah you're right it's similar to the microkernel discussion. You could develop these properties for a shared-kernel container runtime... By making it a microkernel.

It's just that isn't a path with any next steps in the real world. The road from Docker to a secure VM platform is rich with reasonable incremental steps forward (virtualization is an essential step but it's still just one of many). The road from Docker to a microkernel is... Rewrite your entire platform and every workload!

Re: Microsandbox: Virtual Machines that feel and perform like containers

#154

Post by transpute »

delusional wrote: post #148
transpute wrote: post #129

Earlier quoted context omitted.

> You cannot build a secure virtualization runtime because underlying it is the VMM There are VMMs (e.g. pKVM in upstream Linux) with small SLoC that are isolated by silicon support for nested virtualization. This can be found on recent Google Pixel phones/tablets with strong isolation of untrusted Debian Arm Linux "Terminal" VM. A similar architecture was shipped a decade ago by Bromium and now on millions of HP bus…

> silicon support for nested virtualization Is there any guarantee that this "silicon support" is any safer than the software? Once we break the software abstraction down far enough it's all just configuring hardware. Conversely, once you start baking significant complexity into hardware (such as strong security boundaries) it would seem like hardware would be subject to exactly the same bugs as software would, excep…

> Is there any guarantee that this "silicon support" is any safer than the software?

Safety and security claims are only meaningful in the context of threat models. As described in the Xen/uXen/AX video, pKVM and AWS Nitro security talks, one goal is to reduce the size, function and complexity of open-source code running at the highest processor privilege levels [1], minimizing dependency on closed firmware/SMM/TrustZone. Nitro moved some functions (e.g. I/O virtualization) to separate processors, e.g. SmartNIC/DPU. Apple used an Arm T2 secure enclave processor for encryption and some I/O paths, when their main processor was still x86. OCP Caliptra RoT requires OSS firmware signed by both the OEM and hyperscaler customer. It's a never-ending process of reducing attack surface, prioritized by business context.

> hardware would be subject to exactly the same bugs as software would, except it will be hard to update of course

Some "hardware" functions can be updated via microcode, which has been used to mitigate speculative execution vulnerabilities, at the cost of performance.

[1] https://en.wikipedia.org/wiki/Protection_ring

[2] https://en.wikipedia.org/wiki/Transient_execution_CPU_vulner...

Re: Microsandbox: Virtual Machines that feel and perform like containers

#155

Post by transpute »

delusional wrote: post #147
tptacek wrote: post #92

Earlier quoted context omitted.

The issue, at least with multitenant workloads, isn't "container vulnerabilities" as such; it's that standard containers are premised on sharing a kernel, which makes every kernel LPE a potential container escape --- there's a long history of those bugs, and they're only rarely flagged as "container escapes"; it's just sort of understood that a kernel LPE is going to break containers.

> it's just sort of understood that a kernel LPE is going to break containers. I think it's generally understood that any sort of kernel LPE can potentially (and therefore is generally considered to) lead to breaking all security boundaries on the local machine, since the kernel contains no internal security boundaries. That includes both containers, but also everything else such a user separation, hardware virtualiz…

> hardware virtualization controlled by the local kernel

In some architectures, kernel LPE does not break platform (L0/EL2) virtualization, https://news.ycombinator.com/item?id=44141164

  L0/EL2  L1/EL1                   

  pKVM    KVM                  
  AX      Hyper-V / Xen / ESX

Re: Microsandbox: Virtual Machines that feel and perform like containers

#156

Post by mike_hearn »

simonw wrote: post #24
mike_hearn wrote: post #14

Earlier quoted context omitted.

Which platforms do you use?

macOS on my laptop, anything that runs in a container for when I deploy things.

If you use macOS then it has a great sandboxing system built in (albeit, undocumented). Anthropic are starting to experiment with using it in Claude Code to eliminate permission prompts. Claude can choose to run commands inside the sandbox, in which case they execute immediately.

I've thought about making one of these for other coding agents. It's not quite as trivial as it looks and I know how to do it, also on Windows, although it seems quite a few coding agents just pretend Windows doesn't exist unfortunately.

Re: Microsandbox: Virtual Machines that feel and perform like containers

#157

Post by simonw »

mike_hearn wrote: post #156
simonw wrote: post #24

Earlier quoted context omitted.

macOS on my laptop, anything that runs in a container for when I deploy things.

If you use macOS then it has a great sandboxing system built in (albeit, undocumented). Anthropic are starting to experiment with using it in Claude Code to eliminate permission prompts. Claude can choose to run commands inside the sandbox, in which case they execute immediately. I've thought about making one of these for other coding agents. It's not quite as trivial as it looks and I know how to do it, also on Wind…

The lack of documentation for that system is so frustrating! Security feature are the one thing where great documentation should be table stakes, otherwise we are left just wildly guessing how to keep our system secure!

I'm also disheartened by how the man pages for some of the macOS sandboxing commands have declared them deprecated for at least the last five years: https://7402.org/blog/2020/macos-sandboxing-of-folder.html

Re: Microsandbox: Virtual Machines that feel and perform like containers

#158

Post by codethief »

Nypro wrote: post #123
codethief wrote: post #121

Earlier quoted context omitted.

Hi appcypher, very cool project! Does the underlying MicroVM feature provide an OCI runtime interface, so that it could be used as a replacement for runc/crun in Docker/Podman?

No. Not yet. Would be nice to have

Thanks for your response!

One more question: What syscalls do I need to have access to in order to run a MicroVM? I'm asking because ideally I'd like to run container workloads inside existing containers (self-hosted GitLab CI runners) whose configuration (including AppArmor) I don't control.

Re: Microsandbox: Virtual Machines that feel and perform like containers

#159

Post by mgerdts »

dataflow wrote: post #49
drewg123 wrote: post #47

Earlier quoted context omitted.

Without any context in terms of what the VM is doing or what VMM software you use, my best guess is that the OS/VMM are pre-allocating memory for the VM. This might involve paging out other processes' memory, which could take some time. I think task manager would tell you if there is a blip of memory usage and paging activity at the time. And I'm sure windows itself has profilers that can tell you what is happening w…

VirtualBox on Windows, primarily. Though I feel like haven't seen other VMs in the past start up a whole ton faster (maybe a somewhat) (ignoring WSL2). Page files are already disabled, there's plenty of free RAM, and it makes no difference how little RAM the guest is allocated (even if it's 256MB). So no, those are not the issues. VirtualBox itself seems to be doing something slow during that time and I don't know wh…

What is your definition of free memory? If the system has read a lot of data, the page cache is probably occupying most of the RAM you consider free. Look at cache and standby counters.

I’ve noticed that windows can only evict data from the page cache at about 5 GB/s. I do not know if this zeros the memory or that would need to be done in the allocation path.

A couple years ago I tracked down a long pause while starting qemu on Linux to it zeroing the 100s of GB of RAM given to the VM as 1 GB huge pages.

These may or may not be big contributors to what you are seeing, depending on the VM’s RAM size.

Re: Microsandbox: Virtual Machines that feel and perform like containers

#160

Post by arjunbajaj »

SwiftyBug wrote: post #62

Kind of almost off-topic: I'm working on a project where I must run possibly untrusted JavaScript code. I want to run it in an isolated environment. This looks like a very nice solution as I could spin up a microsandbox and securely run the code. I could even have a pool os live sandboxes so I wouldn't even experience the 200ms starts. Because this is OCI-compatible, I could even provide a whole sandboxed environment…

I recommend trying Javy[0]. Javy allows you to build a WASM file that includes Javy's JS interpreter along with your JS source code. Note that Javy is a heavily sandboxed environment so it doesn't have access to the internet, or npm modules, a desirable feature for running user code.

We're building an IoT Cloud Platform, Fostrom[1] where we're using Javy to power our Actions infrastructure. But instead of compiling each Action's JS code to a Javy WASM module, I figured out a simpler way by creating a single WASM module with our wrapper code (which contains some further isolation and helpful functions), and we provide the user code as an input while executing the single pre-compiled WASM module.

[0] https://github.com/bytecodealliance/javy

[1] https://fostrom.io

Post reply on HN