At this point, why not just use a virtual machine? We've come full circle!
GKE Sandbox: Independent operating system kernel to each container
41–50 of 88 posts
Re: GKE Sandbox: Independent operating system kernel to each container
#42Isn't this server-side react rendering? What are we doing? We started with virtual machines and then thought, no, we can share a kernel and do this without the overhead. Now we want each of our containers to have their own kernel. This is full curcle... why not just fire up a VM? Am I missing something? Firecracker doesn't have the product vision behind it to do this, but at some point we will have a microvm technolo…
One reason I can see is the same reason that linuxkit [1] is a really interesting way of putting together Linux OS images: the container ecosystem has produced some tools that are really useful for building and packaging up Linux userspaces, and being able to reuse those tools in other ways is valuable.
With linuxkit you give up some of the niceties of image layer caching, but with an approach like this you get the best of both worlds -- the isolation of VMs but the tooling and usability of containers.
Re: GKE Sandbox: Independent operating system kernel to each container
#43Earlier quoted context omitted.
(I am co-author of the post) System calls are important, but only one factor. The linked doc is an attempt to clarify and delineate various costs. There are number of platform options (the platform is what does syscall interception), and I don't believe any of them are 100x so to say "at least" is a bit disingenuous. You may have confused the "runsc-kvm" number with "using a VM". "runsc-kvm" is the system call perfor…
You're right, that's not the chart I wanted to see. I'm just dubious that reimplementing lots of the Linux kernel in Go while paying the cost of the ptrace interception is worthwhile. It seems like you're just adding a lot of attack surface (admittedly managed code > native code) with a large perf impact. Do you have any docs on how the kvm-runsc platform works? Skimming the files, I don't see some of the bits necess…
I'm not sure what you mean by KVM syscall trapping for the guest. The bluepill refers to the fact that the Sentry runs transparently in VMX non-root ring 0 and regular host ring 3.
I'm not sure what to provide re: docs -- the code is all there, reasonably documented and there are discussions on the public groups of how the KVM platform works. I feel a bit like you're coming in with a specific set of ideas and skimming files (e.g. the performance guide and the code itself) in order to confirm an existing understanding, but it's just not working.
I'd love more precise criticisms re: adding to the attack surface, but otherwise I'm not sure how I can help.
Re: GKE Sandbox: Independent operating system kernel to each container
#44Isn't this server-side react rendering? What are we doing? We started with virtual machines and then thought, no, we can share a kernel and do this without the overhead. Now we want each of our containers to have their own kernel. This is full curcle... why not just fire up a VM? Am I missing something? Firecracker doesn't have the product vision behind it to do this, but at some point we will have a microvm technolo…
(I'm a co-author of the blog post)
Many functions of the kernel are still effectively shared: memory management (e.g. reclaim, swap), thread scheduling, etc. The application is simply limited in its ability to interact with the shared kernel, and functionality related to system APIs is isolated. Arguably I think this is closer to the ergonomics of containers, but with compatibility and performance trade-offs.
Re: GKE Sandbox: Independent operating system kernel to each container
#45Sandboxed containers with kernels - so what's the difference now between this and fully isolated virtual machine?
Another approach might be to make virtual machine technology more like containers. Then the two shall meet.
Re: GKE Sandbox: Independent operating system kernel to each container
#46Earlier quoted context omitted.
You're right, that's not the chart I wanted to see. I'm just dubious that reimplementing lots of the Linux kernel in Go while paying the cost of the ptrace interception is worthwhile. It seems like you're just adding a lot of attack surface (admittedly managed code > native code) with a large perf impact. Do you have any docs on how the kvm-runsc platform works? Skimming the files, I don't see some of the bits necess…
I'm not sure what you mean by KVM syscall trapping for the guest. The bluepill refers to the fact that the Sentry runs transparently in VMX non-root ring 0 and regular host ring 3. I'm not sure what to provide re: docs -- the code is all there, reasonably documented and there are discussions on the public groups of how the KVM platform works. I feel a bit like you're coming in with a specific set of ideas and skimmin…
I'm very skeptical about the platform and don't have the time to devote to reading the codebase or having conversations as I would like. The TL;DR is that the syscall interception technique seems expensive and I wonder if you will write all sorts of logic bugs in the sentry broker. It seems like you folks care about security, and have some good ideas, but if you really care about hostile multi-tenant containers, why not stick the container in a VM and call it a day?
Re: GKE Sandbox: Independent operating system kernel to each container
#47I have read that all containers at Google run inside of a VM and indeed that article mentions that gVisor is in use in things like App Engine and their internal workloads.
So if containers on GKE were already being spun up inside lightweights VMs what does allowing customer's to select the gVisor runtime offer beyond whatever Google's existing lightweight VM already provides?
Re: GKE Sandbox: Independent operating system kernel to each container
#48Isn't this server-side react rendering? What are we doing? We started with virtual machines and then thought, no, we can share a kernel and do this without the overhead. Now we want each of our containers to have their own kernel. This is full curcle... why not just fire up a VM? Am I missing something? Firecracker doesn't have the product vision behind it to do this, but at some point we will have a microvm technolo…
(I work at AWS)
Have you seen https://github.com/firecracker-microvm/firecracker-container... ? The team here is working to make firecracker as seamless as possible for running containerized applications in a microvm.
Re: GKE Sandbox: Independent operating system kernel to each container
#49Earlier quoted context omitted.
The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.
> Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?
Cloud Run/App Engine PM
Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs.
Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.
Re: GKE Sandbox: Independent operating system kernel to each container
#50At this point, why not just use a virtual machine? We've come full circle!
Mostly the performance characteristics. A virtual machine presenting as a machine needs an operating system to be useful. Most operating systems have long-engrained assumptions about the nature of the world, such as:
"There is a time when I go from power-off to power-on, and it is rare, so I may perform expensive operations then to amortise their cost over running time".
or
"While running, time does not skip and hardware does not change".
The practical upshot being that the OS needs to be booted from scratch in a number of scenarios.
But it's not the OS that provides value. It's a means to an end, and that end is to run software. Most software written to run on OSes also have engrained assumptions, such as "I will come to be launched on a fully-booted system".
Containers move the virtualisation up from hardware to the OS API surface. Because the cost of booting is now amortised over all containers running on the system, the original assumptions of both OS designers and software designers become, approximately, true again.
So you're right, we came full circle, but not to a point that means "use fully-dressed VMs again".