Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GKE Sandbox: Independent operating system kernel to each container

cloud.google.com

Post reply on HN
Print view View original

21–30 of 88 posts

Re: GKE Sandbox: Independent operating system kernel to each container

#21

Post by lima »

WestCoastJustin wrote: post #12
roryrjb wrote: post #8

Earlier quoted context omitted.

Isn't this the point of seccomp on Linux and pledge on OpenBSD (and others I'm sure I'm just more familiar with these two), but without this much overhead? Also I'd be interested to know, based on this quote in the post "There’s a saying among security experts: containers do not contain" how Solaris/illumos' Zones and FreeBSD Jails compare.

> re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1].…

> Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything.

That one was an extremely obvious misconfiguration - running with --privileged=true. There dozens of ways to abuse that, probably much easier than using a custom kernel module.

Yes, containers do contain, but the attack surface is MUCH larger than a virtual machine or something like gVisor. Just look at the constant stream of Linux local privilege escalations.

Re: GKE Sandbox: Independent operating system kernel to each container

#22

Post by WestCoastJustin »

lima wrote: post #21
WestCoastJustin wrote: post #12

Earlier quoted context omitted.

> re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1].…

> Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. That one was an extremely obvious misconfiguration - running with --privileged…

Well, it happened and on a pretty popular site too. So if they got it wrong how many other people do. This is a core reason folk should check out gVisor. Not sure why the downvotes as this is a pretty good example use-case?

Re: GKE Sandbox: Independent operating system kernel to each container

#23

Post by longtermsec »

raesene9 wrote: post #18
longtermsec wrote: post #16

Earlier quoted context omitted.

so a) and b) are common in practice. these were not obscure boundary conditions or a corner case. and it was very trivial to exploit. "all software has vulns" is a slippery slope is my overarching point. you can't use that to say that the the security risks and isolation are comparable to gvisor. gvisor does away with a very significant amount of attack surface in the linux kernel and reimplements it in golang, which…

The original point I was making what that dismissing container isolation with the trope "containers don't contain" is overly simplistic, not that I thought that docker/runc containers with a default profile had as small an attack surface as gVisor. Generally the security of a piece of software isn't considered fundamentally flawed just because it has a security bug, otherwise pretty much every piece of software would…

I'm not sure that it is overly simplistic, I think the statement that "containers do not contain" is an intentional oxymoron that points to some ground truths. These ground truths are that a process in a container is running in the same kernel, and although namespaces are meant to isolate some set of resources from other processes, and there are still very many shared resources that might not be isolated at all. This means a lot of attack surface, and exploiting the kernel will grant access to the other processes on the system.

In terms of quantity, 4 is not an accurate picture. I haven't sat down to analyze CVEs (https://www.cvedetails.com/product/47/Linux-Linux-Kernel.htm...), but say out of 50 practically exploitable kernel memory corruption bugs/year 4-5 new bugs every year are reachable from some common namespace configuration for a container. And this just marks what is publicly disclosed, which is a subset of the vulnerabilities attackers know about.

Bounties arent the only outlet for these, see: VEP.

Re: GKE Sandbox: Independent operating system kernel to each container

#24

Post by 013a »

WestCoastJustin wrote: post #12
roryrjb wrote: post #8

Earlier quoted context omitted.

Isn't this the point of seccomp on Linux and pledge on OpenBSD (and others I'm sure I'm just more familiar with these two), but without this much overhead? Also I'd be interested to know, based on this quote in the post "There’s a saying among security experts: containers do not contain" how Solaris/illumos' Zones and FreeBSD Jails compare.

> re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1].…

That's fair, but at the same time: If the end-state is "containers should contain, they're secure, any insecurities are bugs" then why do we see so many defense-in-depth strategies like gVisor pop up which provide legitimate value to consumers?

At what point are we just reinventing the VM hypervisor, but worse because every single one of these systems already has a VM hypervisor running somewhere? It seems likely to me that in the not-so-distant future the "Container" terminology won't actually mean anything because we'll figure out the engineering difficulty behind merging the best parts of VMs with the best parts of Containers, and managed systems like Fargate or even GKE don't really need both a VM hypervisor and a Container hypervisor when they're so similar.

Re: GKE Sandbox: Independent operating system kernel to each container

#25

Post by lima »

013a wrote: post #24
WestCoastJustin wrote: post #12

Earlier quoted context omitted.

> re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1].…

That's fair, but at the same time: If the end-state is "containers should contain, they're secure, any insecurities are bugs" then why do we see so many defense-in-depth strategies like gVisor pop up which provide legitimate value to consumers? At what point are we just reinventing the VM hypervisor, but worse because every single one of these systems already has a VM hypervisor running somewhere? It seems likely to…

gVisor is a special kind of hypervisor, basically - it has a production-ready KVM backend.

The main difficulties with VM-backend containers are storage passthrough and memory overcommit.

Re: GKE Sandbox: Independent operating system kernel to each container

#26

Post by lima »

WestCoastJustin wrote: post #22
lima wrote: post #21

Earlier quoted context omitted.

> Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. That one was an extremely obvious misconfiguration - running with --privileged…

Well, it happened and on a pretty popular site too. So if they got it wrong how many other people do. This is a core reason folk should check out gVisor. Not sure why the downvotes as this is a pretty good example use-case?

gVisor has unsafe modes of operation, too. What I'm saying is that this is not a good example of "Container breakout", as it was just a misconfiguration, not an exploit.

"people are extremely creative and will find ways around everything" is not an excuse - it's a matter of risk management and threat modelling.

Escaping from a VM or gVisor is much, much harder than escaping from Linux namespaces ("containers") due to the MUCH smaller amount of attack surface/amount of exposed code. Using Linux containers in an untrusted multi-tenant environment is very dangerous, especially if you're a high profile cloud provider, which is why all of these projects exist.

Re: GKE Sandbox: Independent operating system kernel to each container

#27

Post by raesene9 »

longtermsec wrote: post #23
raesene9 wrote: post #18

Earlier quoted context omitted.

The original point I was making what that dismissing container isolation with the trope "containers don't contain" is overly simplistic, not that I thought that docker/runc containers with a default profile had as small an attack surface as gVisor. Generally the security of a piece of software isn't considered fundamentally flawed just because it has a security bug, otherwise pretty much every piece of software would…

I'm not sure that it is overly simplistic, I think the statement that "containers do not contain" is an intentional oxymoron that points to some ground truths. These ground truths are that a process in a container is running in the same kernel, and although namespaces are meant to isolate some set of resources from other processes, and there are still very many shared resources that might not be isolated at all. This…

So (as I'm sure you know) linux container isolation isn't just a product of namespaces, but namespaces+capabilities+cgroups+(SELinux/Apparmor)+seccomp-bpf. Each one of those layers provides some aspect of isolation and for a Linux kernel exploit to succeed in escaping a container it needs to bypass/compromise each one (or as in the case of the runc vulnerability occur prior to the sandbox being fully established).

So just taking Linux kernel bugs as a metric doesn't really apply.

That's why I gave the list I did, as those are the only ones which I'm aware of which can bypass all the layers of isolation in a standard Linux container.

If the ground truth "containers don't contain" applies, then it appears you're saying that Linux is innately and architecturally unsuitable for multi-user/process use, which seems like a fairly bold statement given its prevalence...

After all, all a container is, is a Linux process with Linux isolation mechanisms applied to it...

Re: GKE Sandbox: Independent operating system kernel to each container

#28

Post by raesene9 »

WestCoastJustin wrote: post #22
lima wrote: post #21

Earlier quoted context omitted.

> Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. That one was an extremely obvious misconfiguration - running with --privileged…

Well, it happened and on a pretty popular site too. So if they got it wrong how many other people do. This is a core reason folk should check out gVisor. Not sure why the downvotes as this is a pretty good example use-case?

So, play with Docker are trying to do something very niche and not something that almost anyone else would try in production, which is running Docker inside Docker, which they do in order to produce the very cool service they do.

Their breach isn't really a good indicator as I can't think of any/many reasons that most companies would try and do that...

Re: GKE Sandbox: Independent operating system kernel to each container

#29

Post by equalunique »

WestCoastJustin wrote: post #2

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on ste…

My introduction to gVisor was the talk by Emma Haruka Iwao at InfoQ NY 2018: https://www.youtube.com/watch?v=Ur0hbW_K66s

I learned of the talk because Brian Cantrill referenced it during his very deep dive into operating systems, C, and Rust given later at the some event: https://www.youtube.com/watch?v=HgtRAbE1nBM

Re: GKE Sandbox: Independent operating system kernel to each container

#30

Post by muricula »

GKE Sandbox/gVisor syscall performance is at least 100x worse than virtualization[0], which is huge. Why shouldn't I just run everything in a VM/lxc container instead? Is it worth proxying everything through your syscall broker when I can just trust my hypervisor to be a security boundary instead? [0]: https://gvisor.dev/docs/architecture_guide/performance/
Post reply on HN