Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We improved the performance of a userspace TCP stack in Go

coder.com

Post reply on HN
Print view View original

61–70 of 133 posts

Re: We improved the performance of a userspace TCP stack in Go

#61

Post by raggi »

pantalaimon wrote: post #10

The obvious question is: How does it compare to the in-Kernel TCP stack?

It's less mature, which shows up in lots of places, such as sometimes having less than ideal defaults (as in buffer sizes shown here), and bugs if you start using more fancy features (which improve over time of course).

This is approximately the case for any alternative IP stack you might pick though, a mature IP stack is a huge undertaking with all the many flavors of enhancements to IP and particularly TCP over the years, the high variance in platform behaviors and configurations and so on.

In general you should only take on a dependency of a lesser-used IP stack if you're willing to retain or train IP experts in house over the long haul, because as is demonstrated here, taking on such a dependency means eventually you'll find a business need for that expertise. If that's way outside of your budget or wheelhouse, it might be worth skipping.

Re: We improved the performance of a userspace TCP stack in Go

#62

Post by Xelynega »

tptacek wrote: post #50
Xelynega wrote: post #49

Earlier quoted context omitted.

You can also sidestep that "quirky OS limitation" by just setting the first unprivileged port to 0(ip_unprivileged_port_start), no need for an new stack. https://www.kernel.org/doc/Documentation/networking/ip-sysct...

You can do that. Random programs cannot. Our CLI, which also does user-mode WireGuard and TCP/IP, doesn't even want to run under sudo. You're seeing the point, now: you want to build interesting network features that work the same way everywhere without demanding that your users be system administrators. Hence: user-mode TCP/IP.

I'm still not seeing the point sadly.

If you're running on a system you don't administrate that has ports under 1024 set as privileged, there's no way(with or without your cli) to have a userspace program receive TCP or UDP packets coming into the kernel from external devices for these ports(unless I'm completely mistaken).

What can you accomplish with "user-mode TCP/IP" that you can't from userspace with system calls?

Re: We improved the performance of a userspace TCP stack in Go

#63

Post by tptacek »

parhamn wrote: post #58
tptacek wrote: post #48

Earlier quoted context omitted.

A TCP/IP stack is not an "implementation of syscalls". The things most netstack users do with netstack have nothing to do with wanting to move the kernel into userland and everything to do with the fact that the kernel features they want to access are either privileged or (in a lot of IP routing cases) not available at all. Netstack (like any user-mode IP stack) allows programs to do things they couldn't otherwise do…

> The gVisor/perf thing is a tendentious argument Interesting to dismiss it as such. The gvisor netstack is a (big) part of gvisor and this article is discussing how the performance of that component was, and could well still be, garbage. These tools bring marginal capability and performance gains, shoved down peoples throat by manufacturing security paranoia. Oh an it all happens to cost you like 10x time, but look…

None of this has anything to do with security paranoia.

Re: We improved the performance of a userspace TCP stack in Go

#64

Post by tptacek »

Xelynega wrote: post #62
tptacek wrote: post #50

Earlier quoted context omitted.

You can do that. Random programs cannot. Our CLI, which also does user-mode WireGuard and TCP/IP, doesn't even want to run under sudo. You're seeing the point, now: you want to build interesting network features that work the same way everywhere without demanding that your users be system administrators. Hence: user-mode TCP/IP.

I'm still not seeing the point sadly. If you're running on a system you don't administrate that has ports under 1024 set as privileged, there's no way(with or without your cli) to have a userspace program receive TCP or UDP packets coming into the kernel from external devices for these ports(unless I'm completely mistaken). What can you accomplish with "user-mode TCP/IP" that you can't from userspace with system call…

You are completely mistaken. You can do literally anything you want with TCP/IP provided you can talk UDP on any port, by running a user-mode TCP/IP stack over WireGuard on that port.

Re: We improved the performance of a userspace TCP stack in Go

#65

Post by Xelynega »

nonameiguess wrote: post #57
Xelynega wrote: post #49

Earlier quoted context omitted.

You can also sidestep that "quirky OS limitation" by just setting the first unprivileged port to 0(ip_unprivileged_port_start), no need for an new stack. https://www.kernel.org/doc/Documentation/networking/ip-sysct...

When they're talking about classified defense networks, the actual restrictions they mean is least privilege and separation of duties. Devs are not admins. They don't get root privilege on their machines. They can't create virtual network interfaces and they also can't change kernel settings. But if you put a full TCP/IP stack in userspace, well, they can run that and do whatever they want with it. To answer the upst…

If you're talking about developer machines, isn't the best(and easiest) solution to just run a VM that you administer so you can create virtual networks?

If you're talking about production machines, a userspace application wouldn't be able to sniff privileged ports without elevated permissions, so I fail to see how this application would let you get around that limitation.

Re: We improved the performance of a userspace TCP stack in Go

#66

Post by raggi »

Xelynega wrote: post #59
syzcowboy99 wrote: post #14

Earlier quoted context omitted.

gVisor's netstack is still much slower than the kernel's (and likely always will be). The goal of this userspace netstack is not to compete with the kernel on performance, but offer an alternative that is more portable and secure.

How is it more portable or secure than an API that's been stable for decades, and getting constant security fixes? I see an explanation in their blog about avoiding TUN devices since they require elevated permissions, but why would you need a TUN device to send data to/from an application? I can't understand what their product does from the marketing material but it doesn't look like it would require constructing raw…

You can have multiple layers of security boundary on most of the customer-exposed surface area, and avoid more risky surface areas in the kernel.

Portable is a bit of a weird word here because for many of us with gray beards the word means architectures, kernels and systems, but I think in this context it tends to more mean "can run just as easily on my macbook as in a cloud container", but in practice the software isn't that portable, as Go isn't that portable - at least not in the context of vs. a niche C "portable network stack" that would build roughly anywhere that there's a working C toolchain, which is almost everywhere.

Constant security fixes for the kernel are a real pain in deployments unless you follow upstream kernels closely. If your business is in shipping Linux runtimes with a high packing density, you really need to find ways to minimize the exposed Linux surface area, or organize to be able to ship kernel upstream updates at an extremely high frequency (relative to normal infrastructure upgrade rates for kernels / mandatory reboots) (and I would not consider kexec safe in this kind of context, at all).

An alternative approach might be firecracker / microvms and so on, but those have their own tradeoffs too. The core point is that you want more than one layer between the host machines and the user code that wants to interact with Linux features.

Re: We improved the performance of a userspace TCP stack in Go

#67

Post by Xelynega »

tptacek wrote: post #64
Xelynega wrote: post #62

Earlier quoted context omitted.

I'm still not seeing the point sadly. If you're running on a system you don't administrate that has ports under 1024 set as privileged, there's no way(with or without your cli) to have a userspace program receive TCP or UDP packets coming into the kernel from external devices for these ports(unless I'm completely mistaken). What can you accomplish with "user-mode TCP/IP" that you can't from userspace with system call…

You are completely mistaken. You can do literally anything you want with TCP/IP provided you can talk UDP on any port, by running a user-mode TCP/IP stack over WireGuard on that port.

I don't think you understand, and based on your reply it doesn't sound like I'm mistaken.

With this CLI I am able to listen for external packets to port 80 from userspace without any elevated permissions and intercept traffic that's going to an application that's bound to that port on the OS?

Edit: I think I understand what you're trying to do, but if I do then traffic is going from the kernel UDP stack to the userland TCP stack, back to the UDP kernel stack. Not sure how that avoids sending the packet to the kernel. If it's to get around the port restrictions, why can you not just use unprivileged ports?

Re: We improved the performance of a userspace TCP stack in Go

#68

Post by raggi »

andrewstuart wrote: post #16

I have a problem right now which is that it’s slow to copy large files from one side of the earth to the other. Is this the basis of a solution to that maybe?

not enough detail here to provide a good answer, but I can tell you explicitly that if you're using SMB you're likely not going to get good performance here even if your network stack is has tons of space to overcome bdp and congestion challenges.

Re: We improved the performance of a userspace TCP stack in Go

#69

Post by tptacek »

Xelynega wrote: post #67
tptacek wrote: post #64

Earlier quoted context omitted.

You are completely mistaken. You can do literally anything you want with TCP/IP provided you can talk UDP on any port, by running a user-mode TCP/IP stack over WireGuard on that port.

I don't think you understand, and based on your reply it doesn't sound like I'm mistaken. With this CLI I am able to listen for external packets to port 80 from userspace without any elevated permissions and intercept traffic that's going to an application that's bound to that port on the OS? Edit: I think I understand what you're trying to do, but if I do then traffic is going from the kernel UDP stack to the userla…

I'm not sure I can make this any simpler for you or easier to get your head around. If it helps: the idea here is giving an invocation of your program its own IP address. It can then do whatever it likes with TCP/IP for that address; its own routing, arbitrary protocols, whatever. The Go standard library makes it extremely easy to integrate. To the OS, it's all just ordinary socket code.

Re: We improved the performance of a userspace TCP stack in Go

#70

Post by tptacek »

Xelynega wrote: post #65
nonameiguess wrote: post #57

Earlier quoted context omitted.

When they're talking about classified defense networks, the actual restrictions they mean is least privilege and separation of duties. Devs are not admins. They don't get root privilege on their machines. They can't create virtual network interfaces and they also can't change kernel settings. But if you put a full TCP/IP stack in userspace, well, they can run that and do whatever they want with it. To answer the upst…

If you're talking about developer machines, isn't the best(and easiest) solution to just run a VM that you administer so you can create virtual networks? If you're talking about production machines, a userspace application wouldn't be able to sniff privileged ports without elevated permissions, so I fail to see how this application would let you get around that limitation.

No, because VMs are expensive and require some base level of system administration to operate, booting them usually requires privilege, and if the only problem you're trying to solve is reliably running (e.g.) Postgres and Redis protocol between your CLI and a server somewhere, it's extreme overkill.
Post reply on HN