Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GKE Sandbox: Independent operating system kernel to each container

cloud.google.com

Post reply on HN
Print view View original

71–80 of 88 posts

Re: GKE Sandbox: Independent operating system kernel to each container

#71

Post by sayhello »

spyspy wrote: post #66
derefr wrote: post #52

Earlier quoted context omitted.

That's the "what", but what's the "why"? Why run these in the main Borg cluster, rather than running them in the (separate, if I'm understanding you) Borg cluster that GCP uses as its substrate? Is it that the GCP Borg cluster is just big enough for GCP's control-plane, and then the rest of GCP is all Borg-less VM hypervisor boxes (running ESXi or what-have-you), so these gVisor-on-Borg workloads wouldn't have anywhe…

likely just due to the age of appengine vs. all other gcp products

Used to work on 2nd gen AppEngine.

I helped ship the runtimes!

Yes, this is due to age. GAE and Run depend on pieces of infrastructure going back a long time.

Re: GKE Sandbox: Independent operating system kernel to each container

#72

Post by edoo »

andrewstuart wrote: post #45

Sandboxed containers with kernels - so what's the difference now between this and fully isolated virtual machine? Another approach might be to make virtual machine technology more like containers. Then the two shall meet.

I didn't dig into the implementation details but the term para-hvm came to mind, not quite para virtual but not quite full hvm. Perhaps if security is a real issue then HVM is the only real choice.

Re: GKE Sandbox: Independent operating system kernel to each container

#73

Post by pjmlp »

wahern wrote: post #67
lima wrote: post #25

Earlier quoted context omitted.

gVisor is a special kind of hypervisor, basically - it has a production-ready KVM backend. The main difficulties with VM-backend containers are storage passthrough and memory overcommit.

Memory overcommit is addressed by virtio memory ballooning ( https://www.linux-kvm.org/page/Projects/auto-ballooning ). Even OpenBSD supports this as both guest and host. For storage, there's already virtio block devices, not to mention PCI passthrough. But if you mean direct file system access, virtio-fs ( https://virtio-fs.gitlab.io/ ) is just about ready to roll. There's still the issue that you're running an enti…

I kind of agree with you, but actually like that gVisor exists and is written in Go.

As it kind of proves a point about systems level software being written in Go.

I would rather see VM/unikernels take off instead.

Re: GKE Sandbox: Independent operating system kernel to each container

#74

Post by blaisio »

WestCoastJustin wrote: post #2

For anyone who hasn't seen this before. There is a pretty good gVisor Architecture Guide that explains how this works pretty well via a few diagrams [1]. Lots more info on these pages too [2, 3]. > gVisor intercepts application system calls and acts as the guest kernel, without the need for translation through virtualized hardware. gVisor may be thought of as either a merged guest kernel and VMM, or as seccomp on ste…

It's basically the same thing as Wine - Wine provides the Windows API and implements it using Linux syscalls. Gvisor implements the Linux API using Linux syscalls, but with an extra authorization layer. I think people are just so gung ho about VMs that they forgot this was possible and easy (I did).

This is also similar to what Microsoft is doing in Windows with the WSL. This is another example of how we're really just in a big technology cycle. Dynamically typed -> statically typed -> dynamically typed; bare metal -> API wrappers -> VMs -> containers -> API wrappers. Soon we'll probably be back to bare metal.

Re: GKE Sandbox: Independent operating system kernel to each container

#75

Post by bogomipz »

yoshiat wrote: post #69
bogomipz wrote: post #68

Earlier quoted context omitted.

Thanks for the explanation, this makes sense yes. >"The container on top of Borg is used for scheduling and management." Is this the "open source node container manager" box on slide 13 then? I'm guessing this is the Borg's version of the kubelet then? https://speakerdeck.com/jbeda/containers-at-scale?slide=13

That's a very old slide :) I "guess" the slide deck was talking about https://cloud.google.com/compute/docs/containers/deploying-c...

I see. So is "the container on top of Borg is used for scheduling and management" the Borg equivalent of the K8S kubelet then?

Re: GKE Sandbox: Independent operating system kernel to each container

#76

Post by fulafel »

gVisor is easy to run in on your local dev laptop Docker too. It's a nice alternative to running Docker in a VM, if you prefer a security boundary between random Docker containers you get off Docker Hub and your host machine.

After you built or downloaded the gVisor single binary to /usr/local/bin/ or wherever, just put a snippet provided in the gVisor README in the Docker settings file ("runtime": {...}), and bob's your uncle.

Re: GKE Sandbox: Independent operating system kernel to each container

#77

Post by nullwasamistake »

The only advantage I see to containers over VM is RAM sharing. Beyond that, hardware VM's are better performing and much more secure.

gVisor is just another flavor of containers that replaces kernel interfaces with a Go shim layer to reduce the attack surface in return for worse performance.

If somebody could hack ram sharing/overcommit into traditional VM's all this container nonsense could be dispensed with. Containers are a virtualization layer just like the old days when we used the JVM to run "safe" applets on client machines. Like the JVM, the attack surface will always be huge and security issues nearly endless.

Re: GKE Sandbox: Independent operating system kernel to each container

#78

Post by mav3rick »

prattmic wrote: post #56
saagarjha wrote: post #51

Earlier quoted context omitted.

Reimplementing system calls is non-trivial, especially ones that have complex interactions with others (for example, the system calls related to process management). How do you prevent errors when translating this, and how do you implement features that ostensibly require calls to the OS anyways?

For sure, implementing Linux is no easy task, and there is no magic bullet. For compatibility testing, we have extensive system call unit tests [1] and also run many open source test suites. Language runtime tests (e.g., Python, Go, etc) are particularly useful. We also perform continuous fuzzing with Syzkaller [2]. > how do you implement features that ostensibly require calls to the OS anyways? gVisor's kernel is a…

How is this different than a nicerUI over a seccomp filter for your container?

Re: GKE Sandbox: Independent operating system kernel to each container

#79

Post by yoshiat »

bogomipz wrote: post #75
yoshiat wrote: post #69

Earlier quoted context omitted.

That's a very old slide :) I "guess" the slide deck was talking about https://cloud.google.com/compute/docs/containers/deploying-c...

I see. So is "the container on top of Borg is used for scheduling and management" the Borg equivalent of the K8S kubelet then?

As you can see from the Borg paper [1] and the name, "borglet" is the most closest component to "kubelet".

[1] https://pdos.csail.mit.edu/6.824/papers/borg.pdf

Re: GKE Sandbox: Independent operating system kernel to each container

#80

Post by wahern »

pjmlp wrote: post #73
wahern wrote: post #67

Earlier quoted context omitted.

Memory overcommit is addressed by virtio memory ballooning ( https://www.linux-kvm.org/page/Projects/auto-ballooning ). Even OpenBSD supports this as both guest and host. For storage, there's already virtio block devices, not to mention PCI passthrough. But if you mean direct file system access, virtio-fs ( https://virtio-fs.gitlab.io/ ) is just about ready to roll. There's still the issue that you're running an enti…

I kind of agree with you, but actually like that gVisor exists and is written in Go. As it kind of proves a point about systems level software being written in Go. I would rather see VM/unikernels take off instead.

I can't find a link to the thread (in 5 minutes of Googling), but IIRC in a thread discussing the 2nd iteration[1] of a patch to fix the recent runc container breakout exploit this year one of the developers responsible for the patch flat out stated that Go was a poor choice for runc and has resulted in too much pain and ugly hacks. For example, because namespaces are per thread in Linux and you can't control how Go threads (goroutines, cgo stacks) migrate across kernel threads, the most basic task of simply creating and entering a namespace is complicated. (Neither Go nor Linux are amenable to providing a mechanism to alleviate the issue.) And then there's the issue of memory management--too much bloat and lack of fine-grained control as compared to a managed memory environment. These things don't usually matter, but when they do matter they really matter. They can become the primary source of complexity.

[1] The original fix was to exec runc from a memfd-backed copy so writing to /proc/self/exe in the container didn't poison the binary outside the container. But the change in memory usage broke some existing workloads in the wild which had low memory resource limits. I think the second iteration used O_TMPFILE on tmpfs, or at least that was what was under discussion.

Post reply on HN