Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GKE Sandbox: Independent operating system kernel to each container

cloud.google.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: GKE Sandbox: Independent operating system kernel to each container

#51

Post by saagarjha »

prattmic wrote: post #13
WestCoastJustin wrote: post #2

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on ste…

> From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). gVisor actually intercepts and implements the system calls in the user-space kernel. Two specific goals of gVisor are that (1) system calls are never simply allowed and passed through to the host kernel, and (2) you don't need to write a policy c…

Reimplementing system calls is non-trivial, especially ones that have complex interactions with others (for example, the system calls related to process management). How do you prevent errors when translating this, and how do you implement features that ostensibly require calls to the OS anyways?

Re: GKE Sandbox: Independent operating system kernel to each container

#52

Post by derefr »

asciimike wrote: post #49
cameronbrown wrote: post #32

Earlier quoted context omitted.

> Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?

Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.

That's the "what", but what's the "why"? Why run these in the main Borg cluster, rather than running them in the (separate, if I'm understanding you) Borg cluster that GCP uses as its substrate?

Is it that the GCP Borg cluster is just big enough for GCP's control-plane, and then the rest of GCP is all Borg-less VM hypervisor boxes (running ESXi or what-have-you), so these gVisor-on-Borg workloads wouldn't have anywhere to "live" in the GCP cluster?

If that is the issue, then I would have (naively) expected the solution to that to be adding a second, GCP-scale data-plane Borg cluster per zone, just for client workloads; rather than inviting these client workloads to co-mingle with Google's own workloads in the non-GCP part of the DC.

Re: GKE Sandbox: Independent operating system kernel to each container

#53

Post by windexh8er »

raesene9 wrote: post #28
WestCoastJustin wrote: post #22

Earlier quoted context omitted.

Well, it happened and on a pretty popular site too. So if they got it wrong how many other people do. This is a core reason folk should check out gVisor. Not sure why the downvotes as this is a pretty good example use-case?

So, play with Docker are trying to do something very niche and not something that almost anyone else would try in production, which is running Docker inside Docker, which they do in order to produce the very cool service they do. Their breach isn't really a good indicator as I can't think of any/many reasons that most companies would try and do that...

> Their breach isn't really a good indicator as I can't think of any/many reasons that most companies would try and do that...

There are a bunch of legitimate reasons to run Docker in Docker. The most obvious is in a build pipeline. For example Jenkins does Docker builds in containers all the time.

Re: GKE Sandbox: Independent operating system kernel to each container

#54

Post by thesandlord »

bogomipz wrote: post #47

I have read that all containers at Google run inside of a VM and indeed that article mentions that gVisor is in use in things like App Engine and their internal workloads. So if containers on GKE were already being spun up inside lightweights VMs what does allowing customer's to select the gVisor runtime offer beyond whatever Google's existing lightweight VM already provides?

Other way around: Everything at Google runs inside a container, including the VMs

gVisor lets you run multiple untrusted workloads on the same VM, in this case a GKE node.

Re: GKE Sandbox: Independent operating system kernel to each container

#55

Post by longtermsec »

raesene9 wrote: post #27
longtermsec wrote: post #23

Earlier quoted context omitted.

I'm not sure that it is overly simplistic, I think the statement that "containers do not contain" is an intentional oxymoron that points to some ground truths. These ground truths are that a process in a container is running in the same kernel, and although namespaces are meant to isolate some set of resources from other processes, and there are still very many shared resources that might not be isolated at all. This…

So (as I'm sure you know) linux container isolation isn't just a product of namespaces, but namespaces+capabilities+cgroups+(SELinux/Apparmor)+seccomp-bpf. Each one of those layers provides some aspect of isolation and for a Linux kernel exploit to succeed in escaping a container it needs to bypass/compromise each one (or as in the case of the runc vulnerability occur prior to the sandbox being fully established). So…

bingo. one should always assume that userland access on a linux box is a short step away from full system privileges and active exploits are ready for use by an attacker.

docker has started adding hardening with SELinux+Seccomp because theres a realisation that the linux kernel bugs keep coming, but this is just a bandaid. the other problem with this approach is that in practice a hardened config is too restrictive for real-user use and has real maintenance cost so most will never use them (as argued by others in this thread for why the gvisor approach is superior). AppArmor is very poorly maintained, buggy, and not a practical solution

Re: GKE Sandbox: Independent operating system kernel to each container

#56

Post by prattmic »

saagarjha wrote: post #51
prattmic wrote: post #13

Earlier quoted context omitted.

> From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). gVisor actually intercepts and implements the system calls in the user-space kernel. Two specific goals of gVisor are that (1) system calls are never simply allowed and passed through to the host kernel, and (2) you don't need to write a policy c…

Reimplementing system calls is non-trivial, especially ones that have complex interactions with others (for example, the system calls related to process management). How do you prevent errors when translating this, and how do you implement features that ostensibly require calls to the OS anyways?

For sure, implementing Linux is no easy task, and there is no magic bullet. For compatibility testing, we have extensive system call unit tests [1] and also run many open source test suites. Language runtime tests (e.g., Python, Go, etc) are particularly useful. We also perform continuous fuzzing with Syzkaller [2].

> how do you implement features that ostensibly require calls to the OS anyways?

gVisor's kernel is a user-space program, so it can and does make system calls to the host OS. Some examples:

* An application blocks trying to read(2) from a pipe. gVisor ultimately implements blocking by waiting on a Go channel. The Go runtime will ultimately implement this with a futex(2) call to the host OS. * An application reads from a file that is ultimately backed by a file on the host (provided by the Gofer [3]). This will result in a pread(2) system call to the host.

The purpose here isn't to avoid the host completely (that's not possible), but to limit exposure to the host. gVisor can implement all the parts of Linux it does on a much smaller subset of host system calls. Anything we don't use is blocked by a second-level seccomp sandbox around the kernel. e.g., the kernel cannot make obscure system calls, or even open files or create sockets on the host (those operations are controlled by an external agent).

[1] https://github.com/google/gvisor/tree/master/test/syscalls/l...

[2] https://github.com/google/syzkaller

[3] https://gvisor.dev/docs/architecture_guide/overview/

Re: GKE Sandbox: Independent operating system kernel to each container

#57

Post by bogomipz »

thesandlord wrote: post #54
bogomipz wrote: post #47

I have read that all containers at Google run inside of a VM and indeed that article mentions that gVisor is in use in things like App Engine and their internal workloads. So if containers on GKE were already being spun up inside lightweights VMs what does allowing customer's to select the gVisor runtime offer beyond whatever Google's existing lightweight VM already provides?

Other way around: Everything at Google runs inside a container, including the VMs gVisor lets you run multiple untrusted workloads on the same VM, in this case a GKE node.

What would running a VM inside a container provide in terms of security and isolation that just running a VM would not?

This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states:

>"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."[1]

Also see slide 13 of Joe Beda's talk from a five years ago shows the container running in a VM not the other way around:

https://speakerdeck.com/jbeda/containers-at-scale?slide=13

[1] https://queue.acm.org/detail.cfm?id=2898444

Re: GKE Sandbox: Independent operating system kernel to each container

#58

Post by tweenagedream »

derefr wrote: post #52
asciimike wrote: post #49

Earlier quoted context omitted.

Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.

That's the "what", but what's the "why"? Why run these in the main Borg cluster, rather than running them in the (separate, if I'm understanding you) Borg cluster that GCP uses as its substrate? Is it that the GCP Borg cluster is just big enough for GCP's control-plane, and then the rest of GCP is all Borg-less VM hypervisor boxes (running ESXi or what-have-you), so these gVisor-on-Borg workloads wouldn't have anywhe…

Isolation is often done in software, Google has invested a lot of effort in making sure the distinct services that they run e.g. Youtube transcoding on the same machine search is running on don't interfere with each other. Whether through cpu constrains or some other priority levels. These are features of borg.

https://scholar.google.com/scholar?lr&ie=UTF-8&oe=UTF-8&q=La...

Re: GKE Sandbox: Independent operating system kernel to each container

#60

Post by dilyevsky »

metta2uall wrote: post #59

I quite like this defense-in-depth approach, but it's disappointing that it will only be available as part of the probably expensive GKE Advanced. I would have thought safety features should be standard..

I think either way control plane is free now?
Post reply on HN