GKE Sandbox: Independent operating system kernel to each container • Hacker News Board

Re: GKE Sandbox: Independent operating system kernel to each container

#61

Post by yoshiat » Thu, May 16, 2019, 2:01 AM UTC

bogomipz wrote: post #57

thesandlord wrote: post #54

Earlier quoted context omitted.

Other way around: Everything at Google runs inside a container, including the VMs gVisor lets you run multiple untrusted workloads on the same VM, in this case a GKE node.

What would running a VM inside a container provide in terms of security and isolation that just running a VM would not? This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states: >"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and…

(Co-author of the post)

The fact that gVisor is being used in multiple services at Google is probably the confusing part. In case of GKE Sandbox, the users here are external and using Cloud (specifically GKE). The target use case is to add defense in depth to their pods running on potentially shared GKE Nodes (VMs) for Multi-Tenancy. Our talk at Next'19 [1] includes a story by one of our customers, which may help understanding the use cases.

[1] https://www.youtube.com/watch?v=TQfc8OlB2sg

Re: GKE Sandbox: Independent operating system kernel to each container

#62

Post by yoshiat » Thu, May 16, 2019, 2:13 AM UTC

muricula wrote: post #46

amscanne wrote: post #43

Earlier quoted context omitted.

I'm not sure what you mean by KVM syscall trapping for the guest. The bluepill refers to the fact that the Sentry runs transparently in VMX non-root ring 0 and regular host ring 3. I'm not sure what to provide re: docs -- the code is all there, reasonably documented and there are discussions on the public groups of how the KVM platform works. I feel a bit like you're coming in with a specific set of ideas and skimmin…

I'm very skeptical about the platform and don't have the time to devote to reading the codebase or having conversations as I would like. The TL;DR is that the syscall interception technique seems expensive and I wonder if you will write all sorts of logic bugs in the sentry broker. It seems like you folks care about security, and have some good ideas, but if you really care about hostile multi-tenant containers, why…

I replied in other comments but our talk at Next'19 [1] includes a story by one of our customers, which may help understand the use cases. In a nutshell, GKE Sandbox should allow sharing the resources of GKE Nodes (VMs) among multiple tenants.

[1] https://www.youtube.com/watch?v=TQfc8OlB2sg

Re: GKE Sandbox: Independent operating system kernel to each container

#63

Post by thesandlord » Thu, May 16, 2019, 2:32 AM UTC

bogomipz wrote: post #57

thesandlord wrote: post #54

Earlier quoted context omitted.

Other way around: Everything at Google runs inside a container, including the VMs gVisor lets you run multiple untrusted workloads on the same VM, in this case a GKE node.

What would running a VM inside a container provide in terms of security and isolation that just running a VM would not? This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states: >"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and…

(I work for GCP)

It looks something like this:

your container -> Compute Engine VM (GKE Node) -> container -> Borg

The container on top of Borg is used for scheduling and management. Joe's talk has a slide on this. As a GCP customer, you never have to worry about this or care about it, as it is an implementation detail.

>"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."

As a GCP customer using GKE, your applications are separated from other GCP customer using VMs.

However, if you want to run your OWN untrusted workloads, then in the past you would have to spin up a separate VM for untrusted workload A and a one VM for untrusted workload B.

This sucks in terms of resource utilization. It would be better in many cases if you could run workload A and B on the same VM. That's where gVisor comes into play.

your untrusted container -> gVisor -> Compute Engine VM (GKE Node) -> container -> Borg

I hope this makes sense!

Re: GKE Sandbox: Independent operating system kernel to each container

#64

Post by bogomipz » Thu, May 16, 2019, 2:53 AM UTC

yoshiat wrote: post #61

bogomipz wrote: post #57

Earlier quoted context omitted.

What would running a VM inside a container provide in terms of security and isolation that just running a VM would not? This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states: >"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and…

(Co-author of the post) The fact that gVisor is being used in multiple services at Google is probably the confusing part. In case of GKE Sandbox, the users here are external and using Cloud (specifically GKE). The target use case is to add defense in depth to their pods running on potentially shared GKE Nodes (VMs) for Multi-Tenancy. Our talk at Next'19 [1] includes a story by one of our customers, which may help und…

Thanks for the link that does make the use case clear i.e multitenancy/SaaS. Am I correct in assuming though that when someone creates a K8S cluster via GKE that the containers that make up their cluster such as the kubelets and masters are all running in VM underneath?

Re: GKE Sandbox: Independent operating system kernel to each container

#65

Post by yoshiat » Thu, May 16, 2019, 3:01 AM UTC

bogomipz wrote: post #64

yoshiat wrote: post #61

Earlier quoted context omitted.

(Co-author of the post) The fact that gVisor is being used in multiple services at Google is probably the confusing part. In case of GKE Sandbox, the users here are external and using Cloud (specifically GKE). The target use case is to add defense in depth to their pods running on potentially shared GKE Nodes (VMs) for Multi-Tenancy. Our talk at Next'19 [1] includes a story by one of our customers, which may help und…

Thanks for the link that does make the use case clear i.e multitenancy/SaaS. Am I correct in assuming though that when someone creates a K8S cluster via GKE that the containers that make up their cluster such as the kubelets and masters are all running in VM underneath?

Yes, exactly. https://cloud.google.com/kubernetes-engine/docs/concepts/clu...

Re: GKE Sandbox: Independent operating system kernel to each container

#66

Post by spyspy » Thu, May 16, 2019, 3:07 AM UTC

derefr wrote: post #52

asciimike wrote: post #49

Earlier quoted context omitted.

Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.

That's the "what", but what's the "why"? Why run these in the main Borg cluster, rather than running them in the (separate, if I'm understanding you) Borg cluster that GCP uses as its substrate? Is it that the GCP Borg cluster is just big enough for GCP's control-plane, and then the rest of GCP is all Borg-less VM hypervisor boxes (running ESXi or what-have-you), so these gVisor-on-Borg workloads wouldn't have anywhe…

likely just due to the age of appengine vs. all other gcp products

Re: GKE Sandbox: Independent operating system kernel to each container

#67

Post by wahern » Thu, May 16, 2019, 3:07 AM UTC

lima wrote: post #25

013a wrote: post #24

Earlier quoted context omitted.

That's fair, but at the same time: If the end-state is "containers should contain, they're secure, any insecurities are bugs" then why do we see so many defense-in-depth strategies like gVisor pop up which provide legitimate value to consumers? At what point are we just reinventing the VM hypervisor, but worse because every single one of these systems already has a VM hypervisor running somewhere? It seems likely to…

gVisor is a special kind of hypervisor, basically - it has a production-ready KVM backend. The main difficulties with VM-backend containers are storage passthrough and memory overcommit.

Memory overcommit is addressed by virtio memory ballooning (https://www.linux-kvm.org/page/Projects/auto-ballooning). Even OpenBSD supports this as both guest and host.

For storage, there's already virtio block devices, not to mention PCI passthrough. But if you mean direct file system access, virtio-fs (https://virtio-fs.gitlab.io/) is just about ready to roll.

There's still the issue that you're running an entire extra kernel. Not sure that's much slower than using Go; it's probably faster if what was described about bouncing on futexes elsethread is true.

gVisor sounds like the kind of solution that makes sense for Google but not something that would survive in the wider community. The concept sounds great, but using Go sounds horrible, though I'm sure Go made prototyping the concept super simple--specifically goroutines reify execution flow in a nice way, but so would stackful coroutines in C or even Rust, which is easy to implement if you don't need to worry about deep recursion.

Re: GKE Sandbox: Independent operating system kernel to each container

#68

Post by bogomipz » Thu, May 16, 2019, 3:25 AM UTC

thesandlord wrote: post #63

bogomipz wrote: post #57

Earlier quoted context omitted.

What would running a VM inside a container provide in terms of security and isolation that just running a VM would not? This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states: >"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and…

(I work for GCP) It looks something like this: your container -> Compute Engine VM (GKE Node) -> container -> Borg The container on top of Borg is used for scheduling and management. Joe's talk has a slide on this. As a GCP customer, you never have to worry about this or care about it, as it is an implementation detail. >"The isolation is not perfect, though: containers cannot prevent interference in resources that t…

Thanks for the explanation, this makes sense yes.

>"The container on top of Borg is used for scheduling and management."

Is this the "open source node container manager" box on slide 13 then? I'm guessing this is the Borg's version of the kubelet then?

https://speakerdeck.com/jbeda/containers-at-scale?slide=13

Re: GKE Sandbox: Independent operating system kernel to each container

#69

Post by yoshiat » Thu, May 16, 2019, 3:46 AM UTC

bogomipz wrote: post #68

thesandlord wrote: post #63

Earlier quoted context omitted.

(I work for GCP) It looks something like this: your container -> Compute Engine VM (GKE Node) -> container -> Borg The container on top of Borg is used for scheduling and management. Joe's talk has a slide on this. As a GCP customer, you never have to worry about this or care about it, as it is an implementation detail. >"The isolation is not perfect, though: containers cannot prevent interference in resources that t…

Thanks for the explanation, this makes sense yes. >"The container on top of Borg is used for scheduling and management." Is this the "open source node container manager" box on slide 13 then? I'm guessing this is the Borg's version of the kubelet then? https://speakerdeck.com/jbeda/containers-at-scale?slide=13

That's a very old slide :) I "guess" the slide deck was talking about https://cloud.google.com/compute/docs/containers/deploying-c...

Re: GKE Sandbox: Independent operating system kernel to each container

#70

Post by jkaplowitz » Thu, May 16, 2019, 3:49 AM UTC

derefr wrote: post #52

asciimike wrote: post #49

Earlier quoted context omitted.

Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.

That's the "what", but what's the "why"? Why run these in the main Borg cluster, rather than running them in the (separate, if I'm understanding you) Borg cluster that GCP uses as its substrate? Is it that the GCP Borg cluster is just big enough for GCP's control-plane, and then the rest of GCP is all Borg-less VM hypervisor boxes (running ESXi or what-have-you), so these gVisor-on-Borg workloads wouldn't have anywhe…

I know nothing about the decisions behind where Cloud Run and GAE run, but even customer GCE VMs run on top of Borg, not just the control plane. GAE predates most or all of GCP, and there weren't separate GCP clusters when it got launched.

(Used to work for Google including the GCP team, but haven't worked for them for over 4 years and I'm not speaking for them now. I'm reasonably sure this is all already public info.)