Live data from Hacker News

We improved the performance of a userspace TCP stack in Go

coder.com

71–80 of 133 posts

Re: We improved the performance of a userspace TCP stack in Go

#71
post #35

Earlier quoted context omitted.

Can you elaborate on your concern? Is the issue that you don't trust gVisor to keep the cloud provider secure?

Providers managed secure shared environments for decades before ultra inefficient wrappers and runtimes like gVisor existed.

No. The providers that did so soundly used virtualization to accomplish this, and a big part of the appeal of K8s is having a much lightweight unit of scheduling than full virtualization. gVisor is a middle ground between full virtualization and shared-kernel multitenant (which has an abysmal security track record).

Re: We improved the performance of a userspace TCP stack in Go

#72
post #52

Earlier quoted context omitted.

If you want to use netstack without Bazel, just use the go branch: https://github.com/google/gvisor/tree/go go get gvisor.dev/gvisor/pkg/tcpip@go The go branch is auto generated with all of the generated code checked in.

hey Ian, long time. Is there any chance y'all could swap out main so that main contains the generated code version? I don't know the status on those export tools these days as I left the company years ago, but if they could sync with a different branch. This would help various folks quite a bit, as for example tsnet users often fall into the trap of trying to do `go get -u`, which then pulls a non-functional gvisor v…

I don't work on gVisor anymore. That said, I think it would be a tough sell. It would be a pretty big breaking change. Also, there is already a problem with people trying to send patches against the go branch and making it the default would make that much worse.

I think the solution is an automatically exported repository at a different path. Kind of (or maybe exactly) like what Tailscale/bradfitz used to maintain.

Re: We improved the performance of a userspace TCP stack in Go

#73
post #41

Earlier quoted context omitted.

If you want to use netstack without Bazel, just use the go branch: https://github.com/google/gvisor/tree/go go get gvisor.dev/gvisor/pkg/tcpip@go The go branch is auto generated with all of the generated code checked in.

I did this once for an experimental project and found it really difficult to keep the version of gVisor I was using up to date, since it seems like the API is extremely volatile. Anyone else had this experience? If so, is there some way around it that I don't know? Or did I just try it at a bad point in the development timeline?

It could be that you happened to find a period of rapid change, but it is also possible that you ran into the issue that raggi mentioned in the sibling comment.

Re: We improved the performance of a userspace TCP stack in Go

#74
post #71
post #35

Earlier quoted context omitted.

Providers managed secure shared environments for decades before ultra inefficient wrappers and runtimes like gVisor existed.

No. The providers that did so soundly used virtualization to accomplish this, and a big part of the appeal of K8s is having a much lightweight unit of scheduling than full virtualization. gVisor is a middle ground between full virtualization and shared-kernel multitenant (which has an abysmal security track record).

Virtualization, lxc, containers (and K8s), etc were solutions to "secure shared environments". And they have an order of magnitude lower performance hit than gvisor does (Google 'cloudrun python startup times' if you're curious on the real impact of this stuff).

Have we proven they're not secure and safe? Have we broken out of containers yet? Heroku was running LXC for years before docker, did they run into major security woes (actual curious)?

If "secured shared environments" is a more specific term meaning "multi user unix environment", I didn't intend to say that.

Though you already mentioned my whole thread is a bit off topic to this post (and I sorta agree) but then baited me with this comment after. I'm happy to drop it and wait for a Gvisor container runtime thread.

Re: We improved the performance of a userspace TCP stack in Go

#75
post #33
post #28

Earlier quoted context omitted.

Are you referring to gVisor the container runtime, or gVisor/netstack, the TCP/IP stack? I see more uptick in netstack. I don't see proliferation of gVisor itself. "Security" is much more salient to gVisor than it is to netstack.

In the issue of abysmal performance on cloud-compute/PaaS Im talking about the container runtime (most Paas is gVisor or Firecracker, no?) cloudrun, DO, modal, etc. But given this article is about improving gvisors userland tcp performance significantly, it seems like the netstack stuff causes major performance losses too. I saw a github link in another top article today https://github.com/misprit7/computerraria wher…

In the context of coder, the userspace TCP overhead should be negligible. Based on https://gvisor.dev/docs/architecture_guide/performance/ and assuming runc is mostly just using the regular kernel networking stack (I think it does, since it mostly just does syscall filtering?) it should be at most a 30% direct TCP performance hit. But in a real application you typically only spend a negligible amount of total time in the TCP stack - the client code, total e2e latency, and server code corresponding to a particular packet will take much more time.

You'll note their node/ruby benchmarks showed a substantially bigger performance hit. That's because the other gvisor sandboxing functionality (general syscall + file I/O) has more of an impact on performance, but also because these are network-processing bound applications (rare) that were still reaching high QPS in absolute terms for their perspective runtimes (do you know many real-world node apps doing 350qps-800qps per instance?).

Because coder is not likely to be bottlenecked by CPU availability for networking, the resource overhead should be inconsequential, and what's really important is the impact on user latency. But that's something likely on the order of 1ms for a roundtrip that is already spending probably 30-50ms at best in transit between client and server (given that coder's server would be running in a datacenter with clients at home or the office), plus the actual application logic overhead which is at best 10ms. And that's very similar to a lot of gvisor netstack use cases which is why it's not as big of a deal as you think it is.

TLDR: For the stuff you'd actually care about (roundtrip latency) in the coder usecase the perf hit of using gvisor netstack should be like 2% at most, and most likely much less. Either way it's small enough to be imperceivable to the actual human using the client.

Re: We improved the performance of a userspace TCP stack in Go

#76
post #74
post #71

Earlier quoted context omitted.

No. The providers that did so soundly used virtualization to accomplish this, and a big part of the appeal of K8s is having a much lightweight unit of scheduling than full virtualization. gVisor is a middle ground between full virtualization and shared-kernel multitenant (which has an abysmal security track record).

Virtualization, lxc, containers (and K8s), etc were solutions to "secure shared environments". And they have an order of magnitude lower performance hit than gvisor does (Google 'cloudrun python startup times' if you're curious on the real impact of this stuff). Have we proven they're not secure and safe? Have we broken out of containers yet? Heroku was running LXC for years before docker, did they run into major sec…

Yes, we have proven that shared-kernel multitenant is unsafe. The best example (though there are many) is the `waitid` LPE; nobody's container lockdown configuration was blocking `waitid`, which is what you'd have had to do to prevent container code from compromising the kernel. The list of Linux LPEs is long, and syzkaller crashes longer stil.

Re: We improved the performance of a userspace TCP stack in Go

#77
post #33

Earlier quoted context omitted.

In the issue of abysmal performance on cloud-compute/PaaS Im talking about the container runtime (most Paas is gVisor or Firecracker, no?) cloudrun, DO, modal, etc. But given this article is about improving gvisors userland tcp performance significantly, it seems like the netstack stuff causes major performance losses too. I saw a github link in another top article today https://github.com/misprit7/computerraria wher…

In the context of coder, the userspace TCP overhead should be negligible. Based on https://gvisor.dev/docs/architecture_guide/performance/ and assuming runc is mostly just using the regular kernel networking stack (I think it does, since it mostly just does syscall filtering?) it should be at most a 30% direct TCP performance hit. But in a real application you typically only spend a negligible amount of total time in…

Are they even using runc/runsc?

Re: We improved the performance of a userspace TCP stack in Go

#78
post #77

Earlier quoted context omitted.

In the context of coder, the userspace TCP overhead should be negligible. Based on https://gvisor.dev/docs/architecture_guide/performance/ and assuming runc is mostly just using the regular kernel networking stack (I think it does, since it mostly just does syscall filtering?) it should be at most a 30% direct TCP performance hit. But in a real application you typically only spend a negligible amount of total time in…

Are they even using runc/runsc?

At coder, no since "gVisor is a container runtime that reimplements the entire Linux ABI (syscalls) in Go, but we only need the networking for our purposes"

but gvisor was using full runsc for the networking benchmarks I linked, and IIUC runc's networking should be sufficiently similar to unsandboxed networking that I believe runscrunc network performance difference should approximate gvisor netstackvanilla kernel networking.

Re: We improved the performance of a userspace TCP stack in Go

#79
post #76
post #74

Earlier quoted context omitted.

Virtualization, lxc, containers (and K8s), etc were solutions to "secure shared environments". And they have an order of magnitude lower performance hit than gvisor does (Google 'cloudrun python startup times' if you're curious on the real impact of this stuff). Have we proven they're not secure and safe? Have we broken out of containers yet? Heroku was running LXC for years before docker, did they run into major sec…

Yes, we have proven that shared-kernel multitenant is unsafe. The best example (though there are many) is the `waitid` LPE; nobody's container lockdown configuration was blocking `waitid`, which is what you'd have had to do to prevent container code from compromising the kernel. The list of Linux LPEs is long, and syzkaller crashes longer stil.

https://news.ycombinator.com/item?id=40591147

So the PaaS providers mentioned in that comment should be assumed to be compromised?

Re: We improved the performance of a userspace TCP stack in Go

#80
post #60

It's great to see this, I know the team went on a long journey through this and the blog makes it almost look shorter and simpler than it was. I'm hoping one day we can all integrate the support for GSO that's been landing in gvisor too, but so far we've (tailscale) not had a chance to look deeply into that yet. It was really effective for our tun and UDP interfaces though.

> one day we can all integrate the support for GSO that's been landing in gvisor

Google engs recently rewrote the GSO bit, but unlike Tailscale, it is only for TCP, though.

Besides, gvisor has had "software" & "hardware" GSO support for as long as I can remember.

Post reply on HN