Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GKE Sandbox: Independent operating system kernel to each container

cloud.google.com

Post reply on HN
Print view View original

11–20 of 88 posts

Re: GKE Sandbox: Independent operating system kernel to each container

#11

Post by WestCoastJustin »

thesandlord wrote: post #10
WestCoastJustin wrote: post #2

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on ste…

> So, Google's using something like this internally too for their own workloads A public example of this is Cloud Run [1, 2] [1] https://news.ycombinator.com/item?id=19616832 [2] https://cloud.google.com/run/docs/reference/container-contra...

Ah, cool, thanks. I didn't know they were running that under the hood. Yeah, I've checked out Cloud Run via a screencast I did on it a few weeks back [1]. I really like the concept and am looking forward to seeing the evolution of it!

[1] https://sysadmincasts.com/episodes/69-cloud-run-with-knative

Re: GKE Sandbox: Independent operating system kernel to each container

#12

Post by WestCoastJustin »

roryrjb wrote: post #8
WestCoastJustin wrote: post #2

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on ste…

Isn't this the point of seccomp on Linux and pledge on OpenBSD (and others I'm sure I'm just more familiar with these two), but without this much overhead? Also I'd be interested to know, based on this quote in the post "There’s a saying among security experts: containers do not contain" how Solaris/illumos' Zones and FreeBSD Jails compare.

> re: seccomp

This thread has a good answer: https://news.ycombinator.com/item?id=16976392

> "containers do not contain"

Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now.

Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything.

[1] https://www.cyberark.com/threat-research-blog/how-i-hacked-p...

Re: GKE Sandbox: Independent operating system kernel to each container

#13

Post by prattmic »

WestCoastJustin wrote: post #2

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on ste…

> From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config).

gVisor actually intercepts and implements the system calls in the user-space kernel. Two specific goals of gVisor are that (1) system calls are never simply allowed and passed through to the host kernel, and (2) you don't need to write a policy configuration for your application; just put your application inside gVisor and go. These are significant differences over simply using something like seccomp on its own (what the architecture guide calls "Rule-based execution").

Some of this is covered in our security model: https://gvisor.dev/docs/architecture_guide/security/#princip...

Re: GKE Sandbox: Independent operating system kernel to each container

#14

Post by amscanne »

raesene9 wrote: post #6

This is a really interesting add-on to GKE and I'm glad to see vendors starting to offer a variety of container runtimes on their platforms. That said, I'm really not a fan of the opening line where it references the old trope of "containers don't contain" The idea that it's trivial to break out of any Docker style container just doesn't reflect reality. Have there been vulns that allow for container breakout, sure t…

(I'm a co-author of the blog post)

I generally agree re: trope, but it's useful because I'm not sure the core idea is widely understood outside security circles. Many people assume that containers provide a strong isolation boundary, and while a break-out is not trivial, providing more isolation in some cases is important, as you allude.

While one option is certainly to provide a locked down policy, monitor the flow of kernel CVEs, and patch constantly, this may not be feasible for many organizations if a) they lack the technical expertise or b) don't know the workloads they're running a priori and can't apply a fixed policy.

So different container runtimes are about providing additional tools for defense-in-depth. (VMs are fantastic tool for this, but it's also nice to have tools that play well in containerized infrastructure other than custom security policies.) None of these tools will be perfect of course, hopefully they can make it easier to improve on the status quo.

Re: contained.af, this is a great example of the workloads problem. If you have a known workload where you can essentially disable all capabilities and access to system resources (e.g. no network), there are many options for securing that workload. They aren't all generalizable.

Re: GKE Sandbox: Independent operating system kernel to each container

#15

Post by WestCoastJustin »

prattmic wrote: post #13
WestCoastJustin wrote: post #2

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on ste…

> From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). gVisor actually intercepts and implements the system calls in the user-space kernel. Two specific goals of gVisor are that (1) system calls are never simply allowed and passed through to the host kernel, and (2) you don't need to write a policy c…

Awesome, thanks. I need to dig into this a little and just run a few demos / labs. This makes sense though. I really like your comments on this thread too (https://news.ycombinator.com/item?id=16976392).

Re: GKE Sandbox: Independent operating system kernel to each container

#16

Post by longtermsec »

raesene9 wrote: post #9
longtermsec wrote: post #7

Earlier quoted context omitted.

The idea that it's trivial to break out of any Docker style container just doesn't reflect reality. -- not just being contrarian here, actually, the reality is that it might be trivial. and it was demonstratively trivial for a long time (see CVE-2019-5736) As for contained.af -- its not a good indicator, it mostly indicates that the reward doesnt meet the market price for demonstrating an escape from a set of hardene…

So the runc vuln, only applied if you were a) running as root in the container and b) hadn't enabled user namespacing. (Also for completeness, it didn't work on RHEL based distros that applied their standard SELinux policy (IIRC)) Also not specifically a Docker vulnerability, it was a runc issue which also affected other Linux containerization software (e.g. lxc) But despite all that, that's just an example of what I…

so a) and b) are common in practice. these were not obscure boundary conditions or a corner case. and it was very trivial to exploit.

"all software has vulns" is a slippery slope is my overarching point. you can't use that to say that the the security risks and isolation are comparable to gvisor. gvisor does away with a very significant amount of attack surface in the linux kernel and reimplements it in golang, which eliminates many bug classes.

for a realistic risk assessment you should consider the linux kernel as a bottomless barrel of memory management bugs, which are exploitable from within a container, whereas gvisor will have a much more finite set of bugs

On our team we've got extensive experience in finding compromises in this area, particularly in kernels, and that is why I am adamant that one should not think what docker provides meets the bar for best practices in a security critical environment. Something like gvisor would much more fit the bill.

Re: GKE Sandbox: Independent operating system kernel to each container

#17

Post by raesene9 »

amscanne wrote: post #14
raesene9 wrote: post #6

This is a really interesting add-on to GKE and I'm glad to see vendors starting to offer a variety of container runtimes on their platforms. That said, I'm really not a fan of the opening line where it references the old trope of "containers don't contain" The idea that it's trivial to break out of any Docker style container just doesn't reflect reality. Have there been vulns that allow for container breakout, sure t…

(I'm a co-author of the blog post) I generally agree re: trope, but it's useful because I'm not sure the core idea is widely understood outside security circles. Many people assume that containers provide a strong isolation boundary, and while a break-out is not trivial, providing more isolation in some cases is important, as you allude. While one option is certainly to provide a locked down policy, monitor the flow…

Oh I'd agree and gVisor provides (IMO) a smaller attack surface than a default runc container.

With that said both options, and indeed hypervisor based isolation, are generally one security flaw away from a breakout vulnerability, so the only difference in that respect is the incidence of those flaws.

My experience of people's expectations of container isolation is perhaps somewhat different to yours, which is what prompted my initial comment.

It's all too common (in my experience) to see container isolation dismissed using that "containers don't contain" trope, and for me that feels frustrating as the real picture is much more nuanced than that.

It's all about choosing the right isolation technology for a) a given workload and b) a given threat model/attack surface.

There are tradeoffs (both in terms of performance, and in terms of flexibility) in replacing the runc layer with a different container runtime. Sometimes those will make sense, other times not so much :)

All that said I'm very excited to see more options here, as it'll give everyone the choice of what mechanism works for them for specific workloads.

Re: GKE Sandbox: Independent operating system kernel to each container

#18

Post by raesene9 »

longtermsec wrote: post #16
raesene9 wrote: post #9

Earlier quoted context omitted.

So the runc vuln, only applied if you were a) running as root in the container and b) hadn't enabled user namespacing. (Also for completeness, it didn't work on RHEL based distros that applied their standard SELinux policy (IIRC)) Also not specifically a Docker vulnerability, it was a runc issue which also affected other Linux containerization software (e.g. lxc) But despite all that, that's just an example of what I…

so a) and b) are common in practice. these were not obscure boundary conditions or a corner case. and it was very trivial to exploit. "all software has vulns" is a slippery slope is my overarching point. you can't use that to say that the the security risks and isolation are comparable to gvisor. gvisor does away with a very significant amount of attack surface in the linux kernel and reimplements it in golang, which…

The original point I was making what that dismissing container isolation with the trope "containers don't contain" is overly simplistic, not that I thought that docker/runc containers with a default profile had as small an attack surface as gVisor.

Generally the security of a piece of software isn't considered fundamentally flawed just because it has a security bug, otherwise pretty much every piece of software would be in that bucket by now. As such dismissing containers using that trope based on a bug which wasn't discovered when the trope was coined (by Dan Walsh IIRC) doesn't seem appropriate.

There have been (AFAICR) three breakouts that would affect a default Docker installation in the last 3-4 years (Dirty C0w, WaitID, and the runc issue). That doesn't feel like a particularly high incidence, and gVisor has had at least one in it's shorter lifespan...

If it's always trivial to breakout of docker/containerd/runc containers as (if I'm understanding you correctly) you appear to be implying and which is what appears to be implied by the trope, then I imagine people will be making good money from bug bounties for a long time as a lot of companies are creating platforms which execute semi or untrusted code in runc containers.

Re: GKE Sandbox: Independent operating system kernel to each container

#19

Post by spyspy »

WestCoastJustin wrote: post #11
thesandlord wrote: post #10

Earlier quoted context omitted.

> So, Google's using something like this internally too for their own workloads A public example of this is Cloud Run [1, 2] [1] https://news.ycombinator.com/item?id=19616832 [2] https://cloud.google.com/run/docs/reference/container-contra...

Ah, cool, thanks. I didn't know they were running that under the hood. Yeah, I've checked out Cloud Run via a screencast I did on it a few weeks back [1]. I really like the concept and am looking forward to seeing the evolution of it! [1] https://sysadmincasts.com/episodes/69-cloud-run-with-knative

The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.

Re: GKE Sandbox: Independent operating system kernel to each container

#20

Post by bradfitz »

WestCoastJustin wrote: post #15
prattmic wrote: post #13

Earlier quoted context omitted.

> From what I understand, basically a user-space program that wraps your container and intercepts all system calls. You can then allow/deny/re-wire them (based on a config). gVisor actually intercepts and implements the system calls in the user-space kernel. Two specific goals of gVisor are that (1) system calls are never simply allowed and passed through to the host kernel, and (2) you don't need to write a policy c…

Awesome, thanks. I need to dig into this a little and just run a few demos / labs. This makes sense though. I really like your comments on this thread too ( https://news.ycombinator.com/item?id=16976392 ).

App Engine now uses gVisor too:

https://cloud.google.com/blog/products/gcp/introducing-app-e...

Post reply on HN