Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GKE Sandbox: Independent operating system kernel to each container

cloud.google.com

Post reply on HN
Print view View original

31–40 of 88 posts

Re: GKE Sandbox: Independent operating system kernel to each container

#31

Post by lifty »

muricula wrote: post #30

GKE Sandbox/gVisor syscall performance is at least 100x worse than virtualization[0], which is huge. Why shouldn't I just run everything in a VM/lxc container instead? Is it worth proxying everything through your syscall broker when I can just trust my hypervisor to be a security boundary instead? [0]: https://gvisor.dev/docs/architecture_guide/performance/

where did you get the 100x number? Haven't seen it on the page you provided. Also, I checked previously gVisor and the performance indeed worse, but nowhere even close to 100x worse.

Re: GKE Sandbox: Independent operating system kernel to each container

#32

Post by cameronbrown »

spyspy wrote: post #19
WestCoastJustin wrote: post #11

Earlier quoted context omitted.

Ah, cool, thanks. I didn't know they were running that under the hood. Yeah, I've checked out Cloud Run via a screencast I did on it a few weeks back [1]. I really like the concept and am looking forward to seeing the evolution of it! [1] https://sysadmincasts.com/episodes/69-cloud-run-with-knative

The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.

> Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra

What's the reasoning behind this?

Re: GKE Sandbox: Independent operating system kernel to each container

#33

Post by danbeaulieu »

muricula wrote: post #30

GKE Sandbox/gVisor syscall performance is at least 100x worse than virtualization[0], which is huge. Why shouldn't I just run everything in a VM/lxc container instead? Is it worth proxying everything through your syscall broker when I can just trust my hypervisor to be a security boundary instead? [0]: https://gvisor.dev/docs/architecture_guide/performance/

These kernels and micro vmms (kata, firecracker, etc) are aimed at workloads that have already been containerized or are being deployed to container orchestration systems.

For some cases, having something that is compatible with kubernetes is worth the performance penalty, especially if your workload isn't syscall heavy.

Re: GKE Sandbox: Independent operating system kernel to each container

#34

Post by amscanne »

muricula wrote: post #30

GKE Sandbox/gVisor syscall performance is at least 100x worse than virtualization[0], which is huge. Why shouldn't I just run everything in a VM/lxc container instead? Is it worth proxying everything through your syscall broker when I can just trust my hypervisor to be a security boundary instead? [0]: https://gvisor.dev/docs/architecture_guide/performance/

(I am co-author of the post)

System calls are important, but only one factor. The linked doc is an attempt to clarify and delineate various costs. There are number of platform options (the platform is what does syscall interception), and I don't believe any of them are 100x so to say "at least" is a bit disingenuous. You may have confused the "runsc-kvm" number with "using a VM". "runsc-kvm" is the system call performance of gVisor using the kvm platform, which is not a full VM [1]. In general the syscall cost in a VM depends entirely on the guest OS, since there is no VMEXIT for this operation.

VMs are a valid choice depending on your workload, and this is providing an additional tool that provides an easy control for containerized infrastructure. You can use what works for you. Native containers certainly work as well, but you'll probably want to consider additional security controls of some form if you're really running untrusted stuff in there.

[1] https://github.com/google/gvisor/tree/master/pkg/sentry/plat...

Re: GKE Sandbox: Independent operating system kernel to each container

#35

Post by muricula »

lifty wrote: post #31
muricula wrote: post #30

GKE Sandbox/gVisor syscall performance is at least 100x worse than virtualization[0], which is huge. Why shouldn't I just run everything in a VM/lxc container instead? Is it worth proxying everything through your syscall broker when I can just trust my hypervisor to be a security boundary instead? [0]: https://gvisor.dev/docs/architecture_guide/performance/

where did you get the 100x number? Haven't seen it on the page you provided. Also, I checked previously gVisor and the performance indeed worse, but nowhere even close to 100x worse.

Look at the syscalls chart. It's hard to tell exactly what the numbers are on the log log scale on the syscalls chart, and it's probably not the graph I want anyway, but it looks like their runsc-kvm platform clocks in at 1k ns/syscall while their ptrace platform looks close to 100k ns/syscall. The fact that it's log log is telling alone.

Re: GKE Sandbox: Independent operating system kernel to each container

#36

Post by lifty »

muricula wrote: post #35
lifty wrote: post #31

Earlier quoted context omitted.

where did you get the 100x number? Haven't seen it on the page you provided. Also, I checked previously gVisor and the performance indeed worse, but nowhere even close to 100x worse.

Look at the syscalls chart. It's hard to tell exactly what the numbers are on the log log scale on the syscalls chart, and it's probably not the graph I want anyway, but it looks like their runsc-kvm platform clocks in at 1k ns/syscall while their ptrace platform looks close to 100k ns/syscall. The fact that it's log log is telling alone.

Missed that one. The gVisor calls are closer to 40k but the difference is indeed big.

Re: GKE Sandbox: Independent operating system kernel to each container

#37

Post by amscanne »

muricula wrote: post #35
lifty wrote: post #31

Earlier quoted context omitted.

where did you get the 100x number? Haven't seen it on the page you provided. Also, I checked previously gVisor and the performance indeed worse, but nowhere even close to 100x worse.

Look at the syscalls chart. It's hard to tell exactly what the numbers are on the log log scale on the syscalls chart, and it's probably not the graph I want anyway, but it looks like their runsc-kvm platform clocks in at 1k ns/syscall while their ptrace platform looks close to 100k ns/syscall. The fact that it's log log is telling alone.

It's just log, not log log. The chart generates from .csv hosted on the site [1] and the benchmark tools are all open.

The ptrace numbers are 20x and the KVM platform is actually lower than the Docker default case (though that doesn't mean everything is faster, as system call time is only one factor). As I note above, I think you're confused about what the KVM platform is -- it's not a VM.

[1] https://gvisor.dev/performance/syscall.csv

Re: GKE Sandbox: Independent operating system kernel to each container

#38

Post by muricula »

amscanne wrote: post #34
muricula wrote: post #30

GKE Sandbox/gVisor syscall performance is at least 100x worse than virtualization[0], which is huge. Why shouldn't I just run everything in a VM/lxc container instead? Is it worth proxying everything through your syscall broker when I can just trust my hypervisor to be a security boundary instead? [0]: https://gvisor.dev/docs/architecture_guide/performance/

(I am co-author of the post) System calls are important, but only one factor. The linked doc is an attempt to clarify and delineate various costs. There are number of platform options (the platform is what does syscall interception), and I don't believe any of them are 100x so to say "at least" is a bit disingenuous. You may have confused the "runsc-kvm" number with "using a VM". "runsc-kvm" is the system call perfor…

You're right, that's not the chart I wanted to see. I'm just dubious that reimplementing lots of the Linux kernel in Go while paying the cost of the ptrace interception is worthwhile. It seems like you're just adding a lot of attack surface (admittedly managed code > native code) with a large perf impact. Do you have any docs on how the kvm-runsc platform works? Skimming the files, I don't see some of the bits necessary for a bluepill style hypervisor, so I'm not sure why parts are named bluepill in there. I also don't see a lot of the linux kernel paravirt vdev code I would expect, and you seem to imply that you're not telling KVM to enable syscall trapping for the guest.

Re: GKE Sandbox: Independent operating system kernel to each container

#39

Post by [deleted] »

cameronbrown wrote: post #32
spyspy wrote: post #19

Earlier quoted context omitted.

The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.

> Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?

[deleted]

Re: GKE Sandbox: Independent operating system kernel to each container

#40

Post by whalesalad »

Isn't this server-side react rendering? What are we doing?

We started with virtual machines and then thought, no, we can share a kernel and do this without the overhead. Now we want each of our containers to have their own kernel. This is full curcle... why not just fire up a VM? Am I missing something?

Firecracker doesn't have the product vision behind it to do this, but at some point we will have a microvm technology with the ergonomics of containers and then we'll be WAY closer to true portability and better security.

Post reply on HN