GKE Sandbox: Independent operating system kernel to each container • Hacker News Board

Re: GKE Sandbox: Independent operating system kernel to each container

#81

Post by raesene9 » Thu, May 16, 2019, 7:56 AM UTC

longtermsec wrote: post #55

raesene9 wrote: post #27

Earlier quoted context omitted.

So (as I'm sure you know) linux container isolation isn't just a product of namespaces, but namespaces+capabilities+cgroups+(SELinux/Apparmor)+seccomp-bpf. Each one of those layers provides some aspect of isolation and for a Linux kernel exploit to succeed in escaping a container it needs to bypass/compromise each one (or as in the case of the runc vulnerability occur prior to the sandbox being fully established). So…

bingo. one should always assume that userland access on a linux box is a short step away from full system privileges and active exploits are ready for use by an attacker. docker has started adding hardening with SELinux+Seccomp because theres a realisation that the linux kernel bugs keep coming, but this is just a bandaid. the other problem with this approach is that in practice a hardened config is too restrictive f…

For me, that comes down to threat model.

Should every organization assume that every attacker has access to Linux 0-days that they can use to privesc on a box?

My opinion is that that's not a realistic assessment for every attacker.

Do some attackers have that? I'm sure they do, but not every company should assume that every attacker will be able to do that.

And all this goes back again to the original point. The trope "containers don't contain" is overly simplistic and not appropriate for every companies threat mode.

Re: GKE Sandbox: Independent operating system kernel to each container

#82

Post by metta2uall » Thu, May 16, 2019, 11:53 AM UTC

dilyevsky wrote: post #60

metta2uall wrote: post #59

I quite like this defense-in-depth approach, but it's disappointing that it will only be available as part of the probably expensive GKE Advanced. I would have thought safety features should be standard..

I think either way control plane is free now?

Well gVisor doesn't use the control plane. It is free, but I wouldn't think it has a high cpu or memory load, and Google would make a lot of profit on the nodes.

Re: GKE Sandbox: Independent operating system kernel to each container

#83

Post by d1zzy » Thu, May 16, 2019, 6:09 PM UTC

wahern wrote: post #67

lima wrote: post #25

Earlier quoted context omitted.

gVisor is a special kind of hypervisor, basically - it has a production-ready KVM backend. The main difficulties with VM-backend containers are storage passthrough and memory overcommit.

Memory overcommit is addressed by virtio memory ballooning ( https://www.linux-kvm.org/page/Projects/auto-ballooning ). Even OpenBSD supports this as both guest and host. For storage, there's already virtio block devices, not to mention PCI passthrough. But if you mean direct file system access, virtio-fs ( https://virtio-fs.gitlab.io/ ) is just about ready to roll. There's still the issue that you're running an enti…

One big problem with KVM based VMs that gVisor fixes is that KVM is a (complex) piece of host kernel software. There have been many security incidents in the past related to KVM and there will be more for sure. With gVisor the "virtualization logic" runs purely in user space (and may itself be further isolated, like any other regular user space process, within the host environment). This means that any bugs in gVisor will, at most, impact the isolation unit where it runs in the host space, as opposed to KVM where bugs in KVM would impact the entire machine (including other customer workloads on that machine).

The non-security related issues you listed, specialized interfaces to allow I/O to bypass the generic hardware virtualization layer, are IMO hacks (even the name of "para-virtualization" given to such mechanisms should be a tell). Because it would be to inefficient to do almost any I/O we care about to perform fast (network and storage) through the overall machine virtualization interface, we poke holes in that interface, specialized ones, that will allow us to carry requests and replies from the guest to the host more directly/efficiently. As a software engineer that seems like a hack. When something like gVisor comes along which provides much better security for the host environment and allows to quickly handle syscall level I/O by design I much prefer that approach over a VM. The drawback of gVisor is one similar to Wine: having to write bug for bug compatibility with the ABI supported (Linux x64 in this case). However, different from Wine, the Linux ABI surface is extremely small vs to what Wine has to reimplement to run even the simplest Windows applications and, most of all, with gVisor there's direct access to the source code of the ABI that it needs to implement making development much easier than something like Wine.

Re: GKE Sandbox: Independent operating system kernel to each container

#84

Post by ryacko » Thu, May 16, 2019, 8:32 PM UTC

longtermsec wrote: post #23

raesene9 wrote: post #18

Earlier quoted context omitted.

The original point I was making what that dismissing container isolation with the trope "containers don't contain" is overly simplistic, not that I thought that docker/runc containers with a default profile had as small an attack surface as gVisor. Generally the security of a piece of software isn't considered fundamentally flawed just because it has a security bug, otherwise pretty much every piece of software would…

I'm not sure that it is overly simplistic, I think the statement that "containers do not contain" is an intentional oxymoron that points to some ground truths. These ground truths are that a process in a container is running in the same kernel, and although namespaces are meant to isolate some set of resources from other processes, and there are still very many shared resources that might not be isolated at all. This…

If you do cybersecurity work and Zerodium bug bounties for your stack are less than your yearly wages, you are honor-bound to offer your resignation and request that the company use your salary towards bug bounties.

Fortunately zerodays aren't commonly used.

Re: GKE Sandbox: Independent operating system kernel to each container

#85

Post by ryacko » Thu, May 16, 2019, 8:43 PM UTC

blaisio wrote: post #74

WestCoastJustin wrote: post #2

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on ste…

It's basically the same thing as Wine - Wine provides the Windows API and implements it using Linux syscalls. Gvisor implements the Linux API using Linux syscalls, but with an extra authorization layer. I think people are just so gung ho about VMs that they forgot this was possible and easy (I did). This is also similar to what Microsoft is doing in Windows with the WSL. This is another example of how we're really ju…

There are web hosts offer Raspberry PIs, but they tend to be more expensive than VMs. I'm guessing colocation costs are dominant.

Re: GKE Sandbox: Independent operating system kernel to each container

#86

Post by dilyevsky » Fri, May 17, 2019, 1:07 AM UTC

metta2uall wrote: post #82

dilyevsky wrote: post #60

Earlier quoted context omitted.

I think either way control plane is free now?

Well gVisor doesn't use the control plane. It is free, but I wouldn't think it has a high cpu or memory load, and Google would make a lot of profit on the nodes.

I know but they may conceivably just charge fixed fee for enabling that option on the nodepool.

> it has a high cpu or memory load, and Google would make a lot of profit on the nodes.

They currently solve that problem by having their node VMs melt down at like 50% utilization so you have to run everything with huge padding.

Re: GKE Sandbox: Independent operating system kernel to each container

#87

Post by ian-lewis » Fri, May 17, 2019, 6:59 AM UTC

wahern wrote: post #67

lima wrote: post #25

Earlier quoted context omitted.

gVisor is a special kind of hypervisor, basically - it has a production-ready KVM backend. The main difficulties with VM-backend containers are storage passthrough and memory overcommit.

Memory overcommit is addressed by virtio memory ballooning ( https://www.linux-kvm.org/page/Projects/auto-ballooning ). Even OpenBSD supports this as both guest and host. For storage, there's already virtio block devices, not to mention PCI passthrough. But if you mean direct file system access, virtio-fs ( https://virtio-fs.gitlab.io/ ) is just about ready to roll. There's still the issue that you're running an enti…

There's a blub on "why Go?" on the website if you're interested.

https://gvisor.dev/docs/architecture_guide/ > gVisor is written in Go in order to avoid security pitfalls that can plague kernels. With Go, there are strong types, built-in bounds checks, no uninitialized variables, no use-after-free, no stack overflow, and a built-in race detector. (The use of Go has its challenges too, and isn’t free.)

Using a memory-safe language was a conscious design decision. https://twitter.com/LazyFishBarrel/status/112900096574140416...

Re: GKE Sandbox: Independent operating system kernel to each container

#88

Post by ian-lewis » Fri, May 17, 2019, 7:09 AM UTC

lima wrote: post #21

WestCoastJustin wrote: post #12

Earlier quoted context omitted.

> re: seccomp This thread has a good answer: https://news.ycombinator.com/item?id=16976392 > "containers do not contain" Is sort of troll bait. They do contain. That is why everyone is using them. Sure, there will be exploits to break out of them, just like with VMs, and even CPU bugs now. Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [1].…

> Here is a good example of someone who broke out of a container on the play-with-docker.com site using a custom kernel module [4]. This allowed a container escape but you could say this was a bug since that wasn't the intent. So, you'd patch it. So, I get the joke in that people are extremely creative and will find ways around everything. That one was an extremely obvious misconfiguration - running with --privileged…

Rather that the contain/don't-contain dichotomy, what's more important is gVisor's design principle that it always has 2 layers of isolation from the host and doesn't rely on any one bug in the Linux kernel, sentry, or elsewhere in order to break out of the sandbox. This leaves you less exposed to 0-day attacks and lags in patching kernels.

You can't get that from normal Linux containers due to their fundamental design.