Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We improved the performance of a userspace TCP stack in Go

coder.com

Post reply on HN
Print view View original

41–50 of 133 posts

Re: We improved the performance of a userspace TCP stack in Go

#41

Post by dave78 »

iangudger wrote: post #31
dpeckett wrote: post #20

Really cool to see others hacking on netstack, bit of a shame it's tied up in the gVisor monorepo (and all the Bazel idiosyncracies) but it's a very neat piece of kit. I've actually been hacking on a similar FOSS project lately, with a focus on building what I'm calling a layer 3 service mesh for the edge. More or less came out of my learned hatred for managing mTLS at scale and my dislike for shoving everything thro…

If you want to use netstack without Bazel, just use the go branch: https://github.com/google/gvisor/tree/go go get gvisor.dev/gvisor/pkg/tcpip@go The go branch is auto generated with all of the generated code checked in.

I did this once for an experimental project and found it really difficult to keep the version of gVisor I was using up to date, since it seems like the API is extremely volatile. Anyone else had this experience? If so, is there some way around it that I don't know? Or did I just try it at a bad point in the development timeline?

Re: We improved the performance of a userspace TCP stack in Go

#42

Post by tptacek »

vlovich123 wrote: post #36
tptacek wrote: post #18

Earlier quoted context omitted.

They're not creating raw sockets†. The neat thing about WireGuard is that it runs over vanilla UDP, and presents to the "client" a full TCP/IP interface. We normally plug that interface directly into the kernel, but you don't have to; you can just write a userspace program that speaks WireGuard directly, and through it give a TCP/IP stack interface directly to your program. † I don't think? I didn't see them say that…

So it tunnels TCP/IP over Wireguard UDP?

Correct (I mean, that's fundamentally what WireGuard is: a UDP TCP/IP tunnel, with strong modern encryption).

Re: We improved the performance of a userspace TCP stack in Go

#43

Post by ignoramous »

dave78 wrote: post #41
iangudger wrote: post #31

Earlier quoted context omitted.

If you want to use netstack without Bazel, just use the go branch: https://github.com/google/gvisor/tree/go go get gvisor.dev/gvisor/pkg/tcpip@go The go branch is auto generated with all of the generated code checked in.

I did this once for an experimental project and found it really difficult to keep the version of gVisor I was using up to date, since it seems like the API is extremely volatile. Anyone else had this experience? If so, is there some way around it that I don't know? Or did I just try it at a bad point in the development timeline?

The API is indeed prone to change without notice, but it isn't anything terribly unmanageable.

> really difficult to keep the version of gVisor I was using up to date

For our project, we update gvisor whenever Tailscale does.

Re: We improved the performance of a userspace TCP stack in Go

#44

Post by ignoramous »

vlovich123 wrote: post #36
tptacek wrote: post #18

Earlier quoted context omitted.

They're not creating raw sockets†. The neat thing about WireGuard is that it runs over vanilla UDP, and presents to the "client" a full TCP/IP interface. We normally plug that interface directly into the kernel, but you don't have to; you can just write a userspace program that speaks WireGuard directly, and through it give a TCP/IP stack interface directly to your program. † I don't think? I didn't see them say that…

So it tunnels TCP/IP over Wireguard UDP?

Yes; also see: https://github.com/WireGuard/wireguard-go/blob/12269c2761734...

Re: We improved the performance of a userspace TCP stack in Go

#45

Post by lima »

parhamn wrote: post #35
loosescrews wrote: post #32

Earlier quoted context omitted.

Can you elaborate on your concern? Is the issue that you don't trust gVisor to keep the cloud provider secure?

Providers managed secure shared environments for decades before ultra inefficient wrappers and runtimes like gVisor existed.

OpenVZ, Virtuozzo and friends definitely weren't secure the way gVisor or Firecracker are. You can still do that and some providers do, doesn't make it a good idea.

Re: We improved the performance of a userspace TCP stack in Go

#46

Post by parhamn »

tptacek wrote: post #40
parhamn wrote: post #33

Earlier quoted context omitted.

In the issue of abysmal performance on cloud-compute/PaaS Im talking about the container runtime (most Paas is gVisor or Firecracker, no?) cloudrun, DO, modal, etc. But given this article is about improving gvisors userland tcp performance significantly, it seems like the netstack stuff causes major performance losses too. I saw a github link in another top article today https://github.com/misprit7/computerraria wher…

I don’t believe many PAAS run gVisor; a surprising number just run multitenant docker. The netstack stuff here has nothing to do with the rest of gVisor.

> The netstack stuff here has nothing to do with the rest of gVisor.

How so? Besides being part of it, it is at least similar in the group of "bloated slow userland implementation of things the kernel handles well"

Re: We improved the performance of a userspace TCP stack in Go

#47

Post by mindcrime »

taeric wrote: post #29
tptacek wrote: post #27

Earlier quoted context omitted.

I understand the impulse, but I think it misconstrues the "red tape" this method avoids. It's sidestepping a quirky OS limitation, which dates back to an era of "privileged ports" and multi-user machines. It's not really sidestepping any sort of modern policy boundary. For instance: you could do the exact same thing with WebSockets (and people do).

I was thinking websockets; though, I thought those largely hit the same criticisms? That is, tons of things moved to them specifically to avoid any firewall rules about what they were allowed to send over a network. I'll fully grant that that seems to be the norm for everything browser related. Policies got difficult to install new software, just point your browser to this url and call it a day.

I was thinking websockets; though, I thought those largely hit the same criticisms? That is, tons of things moved to them specifically to avoid any firewall rules about what they were allowed to send over a network.

Arguably, this basic phenomenon has been going on for 20+ years. A lot of people by 2005-2007 or so had come to belive (and probably correctly) that a lot of the impetus for adopting SOAP based web-services over the preceding few years was simply because everything ran over ports 80 and 443 which were already open in the firewall. So deploying a remote service this way was more tractable than submitting a request to allow access to yet another port in firewall, and deal with the inevitable bureaucratic nightmare of getting that approved.

Re: We improved the performance of a userspace TCP stack in Go

#48

Post by tptacek »

parhamn wrote: post #46
tptacek wrote: post #40

Earlier quoted context omitted.

I don’t believe many PAAS run gVisor; a surprising number just run multitenant docker. The netstack stuff here has nothing to do with the rest of gVisor.

> The netstack stuff here has nothing to do with the rest of gVisor. How so? Besides being part of it, it is at least similar in the group of "bloated slow userland implementation of things the kernel handles well"

A TCP/IP stack is not an "implementation of syscalls". The things most netstack users do with netstack have nothing to do with wanting to move the kernel into userland and everything to do with the fact that the kernel features they want to access are either privileged or (in a lot of IP routing cases) not available at all. Netstack (like any user-mode IP stack) allows programs to do things they couldn't otherwise do at all.

The gVisor/perf thing is a tendentious argument. You can have whatever opinion you like about whether running a platform under gVisor supervision is a good idea. But the post we're commenting on is obviously not about gVisor; it's about a library inside of gVisor that is probably a lot more popular than gVisor itself.

Re: We improved the performance of a userspace TCP stack in Go

#49

Post by Xelynega »

tptacek wrote: post #27
taeric wrote: post #24

Earlier quoted context omitted.

I think the point was more that doing this as a way to avoid the red tape of getting permission to open a new connection is odd?

I understand the impulse, but I think it misconstrues the "red tape" this method avoids. It's sidestepping a quirky OS limitation, which dates back to an era of "privileged ports" and multi-user machines. It's not really sidestepping any sort of modern policy boundary. For instance: you could do the exact same thing with WebSockets (and people do).

You can also sidestep that "quirky OS limitation" by just setting the first unprivileged port to 0(ip_unprivileged_port_start), no need for an new stack.

https://www.kernel.org/doc/Documentation/networking/ip-sysct...

Re: We improved the performance of a userspace TCP stack in Go

#50

Post by tptacek »

Xelynega wrote: post #49
tptacek wrote: post #27

Earlier quoted context omitted.

I understand the impulse, but I think it misconstrues the "red tape" this method avoids. It's sidestepping a quirky OS limitation, which dates back to an era of "privileged ports" and multi-user machines. It's not really sidestepping any sort of modern policy boundary. For instance: you could do the exact same thing with WebSockets (and people do).

You can also sidestep that "quirky OS limitation" by just setting the first unprivileged port to 0(ip_unprivileged_port_start), no need for an new stack. https://www.kernel.org/doc/Documentation/networking/ip-sysct...

You can do that. Random programs cannot. Our CLI, which also does user-mode WireGuard and TCP/IP, doesn't even want to run under sudo. You're seeing the point, now: you want to build interesting network features that work the same way everywhere without demanding that your users be system administrators. Hence: user-mode TCP/IP.
Post reply on HN