Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux network performance parameters

github.com

Post reply on HN
Print view View original

51–60 of 117 posts

Re: Linux network performance parameters

#51

Post by jeffbee »

tptacek wrote: post #45
jeffbee wrote: post #39

Earlier quoted context omitted.

"and Pony Express" is the operative phrase. As the paper states on page 1, "Snap is deployed to over half of our fleet of machines and supports the needs of numerous teams." According to the paper it is not niche.

Isn't Pony Express a ground-up replacement for all of TCP/IP? It doesn't even present a TCP/UDP socket interface.

Correct. That is my point. The sockets interface, and design choices within the Linux kernel, make ordinary TCP sockets too difficult to exploit in a datacenter environment. The general trend is away from TCP sockets. QUIC (HTTP/3) is a less extreme retreat from TCP, moving all the flow control, congestion, and retry logic out of the kernel and into the application.

An example of how Linux TCP is unsuitable for datacenters is that the minimum RTO is hard-coded to 200ms, which is essentially forever. People have been trying to land better or at least more configurable parameters upstream for decades. I am hardly the first person to point out the deficiencies. Google presented tuning Linux for datacenter applications at LPC 2022, and their deck has barely changed in 15 years.

Re: Linux network performance parameters

#52

Post by tptacek »

jeffbee wrote: post #51
tptacek wrote: post #45

Earlier quoted context omitted.

Isn't Pony Express a ground-up replacement for all of TCP/IP? It doesn't even present a TCP/UDP socket interface.

Correct. That is my point. The sockets interface, and design choices within the Linux kernel, make ordinary TCP sockets too difficult to exploit in a datacenter environment. The general trend is away from TCP sockets. QUIC (HTTP/3) is a less extreme retreat from TCP, moving all the flow control, congestion, and retry logic out of the kernel and into the application. An example of how Linux TCP is unsuitable for datac…

At the point where we're talking about applications that don't even use standard protocols, we've stopped supplying data points about whether FreeBSD's stack is faster than Linux's, which is the point of the thread.

Later

Also, the idea that QUIC is a concession made to intractable Linux stack problems (the subtext I got from that comment) seems pretty off, since the problems QUIC addresses (HOLB, &c) are old, well known, and were the subject of previous attempts at new transports (SCTP, notably).

Re: Linux network performance parameters

#53

Post by patmorgan23 »

8K832d7tNmiQ wrote: post #15
freedomben wrote: post #7

Could anyone recommend a video or video series covering similar material? There lots on networking in general, but I've had a hard time finding some on Linux specific implementation

I'm also seconding this, but from microcontroller perspective. I want to try developing a simple tcp echo server for a microcontroller, but most examples just use the vendor's own tcp library and put no effort explaining how to manually setup and establish connection to the router.

Well you can always read the standard

Re: Linux network performance parameters

#54

Post by doctorpangloss »

jeffbee wrote: post #21
doctorpangloss wrote: post #16

Does performance tuning for Wi-Fi adapters matter? On desktops, other than disabling features, can anything fix the problems with i210 and i225 ethernet? Those seem to be the two most common NICs nowadays. I don't really understand why common networking hardware and drivers are so flawed. There is a lot of attention paid to RISC-V. How about start with a fully open and correct NIC? They'll shove it in there if it's c…

i225 is just broken but I get excellent performance from i210. 1gb is hardly challenging on a contemporaneous CPU, and the i210 offers 4 queues. What's your beef with i210?

There are a lot of problems with the i210. Here’s a sample:

https://www.google.com/search?q=i210+proxmox+e1000e+disable

Most people don’t really use their NICs “all the time” “with many hosts.” The i210 in particular will hang after a few months of e.g. etcd cluster traffic on 9th and 10th gen Intel, which is common for SFFPCs.

On Windows, the ndis driver works a lot better. Many disconnects in similar traffic load as Linux, and features like receive side coalescing are broken. They also don’t provide proper INFs for Windows server editions, just because.

I assume Intel does all of this on purpose. I don’t think their functionally equivalent server SKUs are this broken.

Apparently the 10Gig patents are expiring very soon. That will make Realtek, Broadcom and Aquantia’s chips a lot cheaper. IMO, motherboards should be much smaller, shipping with BMC and way more rational IO: SFP+, 22110, Oculink, U.2, and PCIe spaced for Infinity Fabric & NVLink. Everyone should be using LVFS for firmware - NVMe firmware, despite being standardized to update, is a complete mess with bugs on every major controller.

I share all of this as someone with experience in operating commodity hardware at scale. People are so wasteful with their hardware.

Re: Linux network performance parameters

#55

Post by elabajaba »

doctorpangloss wrote: post #16

Does performance tuning for Wi-Fi adapters matter? On desktops, other than disabling features, can anything fix the problems with i210 and i225 ethernet? Those seem to be the two most common NICs nowadays. I don't really understand why common networking hardware and drivers are so flawed. There is a lot of attention paid to RISC-V. How about start with a fully open and correct NIC? They'll shove it in there if it's c…

> Does performance tuning for Wi-Fi adapters matter?

If you're willing to potentially sacrifice 10-20% of (max local network) throughput you can drastically improve wifi fairness and improve ping times/reduce bufferbloat (random ping spikes will still happen on wifi though).

There's a huge thread https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/590... that has stuff about enabling and tuning aqm, and some of the tradeoffs between throughput and latency.

Re: Linux network performance parameters

#56

Post by pengaru »

klabb3 wrote: post #49

A random thing I ran into with the defaults (Ubuntu Linux): - net.ipv4.tcp_rmem ~ 6MB - net.core.rmem_max ~ 1MB So.. the tcp_rmem value overrides by default, meaning that the TCP receive window for a vanilla TCP socket actually goes up to 6MB if needed (in reality - 3MB because of the halving, but let's ignore that for now since it's a constant). But if I "setsockopt SO_RCVBUF" in a user-space application, I'm actual…

net.ipv4.tcp_rmem max is a limit for the auto-tuning the kernel performs

once you do SO_RCVBUF the auto-tuning is out of the picture for that socket, and net.core.rmem_max becomes the max.

It's pretty clearly documented @ Documentation/networking/ip-sysctl.rst

Edit: downvotes, really? smh

Re: Linux network performance parameters

#57

Post by drewg123 »

napkin wrote: post #50

Just changing Linux's default congestion control (net.ipv4.tcp_congestion_control) to 'bbr' can make a _huge_ difference in some scenarios, I guess over distances with sporadic packet loss and jitter, and encapsulation. Over the last year, I was troubleshooting issues with the following connection flow: client host reverse proxy host service host On average, I could not get better than 20% theoretical max throughput.…

The difference is that BBR does not use loss as a signal of congestion. Most TCP stacks will cut their send windows in half (or otherwise greatly reduce them) at the first sign of loss. So if you're on a lossy VPN, or sending a huge burst at 1Gb/s on a 10Mb/s VPN uplink, TCP will normally see loss, and back way off.

BBR tries to find Bottleneck Bandwidth rate. Eg, the bandwidth of the narrowest or most congested link. It does this by measuring the round trip time, and increasing the transmit rate until the RTT increases. When the RTT increases, the assumption is that a queue is building at the narrowest portion of the path and the increase of RTT is proportional to the queue depth. It then drops rate until the RTT normalizes due to the queue draining. It sends at that rate for a period of time, and then slightly increases the rate to see if RTT increases again (if not, it means that the queuing that saw before was due to competing traffic which has cleared).

I upgraded from a 10Mb/s cable uplink to 1Gb/s symmetrical fiber a few years ago. When I did so, I was ticked that my upload speed on my corp. VPN remained at 5Mb/s or so. When I switched to RACK TCP (or BBR) on FreeBSD, my upload went up by a factor of 8 or so, to about 40Mb/s, which is the limit of the VPN.

Re: Linux network performance parameters

#58

Post by devonkim »

nolist_policy wrote: post #47
jeffbee wrote: post #39

Earlier quoted context omitted.

"and Pony Express" is the operative phrase. As the paper states on page 1, "Snap is deployed to over half of our fleet of machines and supports the needs of numerous teams." According to the paper it is not niche.

Makes sense, they're probably using QUIC in lots of products and the kernel can't accelerate that anyways, it would only pass opaque UDP packets to and from the application.

Last I remember as of at least 7 years ago Google et al were using custom NIC firmware to avoid having the kernel get involved in general (I think they managed to do a lot of Maglev directly on the NICs) because latency is so dang important at high speed networking speeds that letting anything context switch and need to wait on the kernel is a big performance hit. Not a lot of room for latency when you're working at 100 Gbps.

Re: Linux network performance parameters

#59

Post by drewg123 »

tptacek wrote: post #30
dijit wrote: post #14

Earlier quoted context omitted.

This is "appeal to authority" fallacy incarnate. Google/Amazon "etc;" are likely happy to pay the cost because it really is "good enough" and the benefits of Linux over FreeBSD are otherwise quite considerable. Google in particular seems blissfully happy to literally throw hardware at problems; since hardware is (for them especially) fundamentally extremely cheap. Even multiple percentage gains in throughput are not…

Do you know for a fact that Google primarily uses userland networking, or does that just seem accurate to you?

Google makes heavy use of userspace networking. I was there roughly a decade ago. At least at that time, a major factor is the choice of userspace over kernel networking was time to deployment. Services like the ones described above were built on the monorepo, and could be deployed in seconds at the touch of a button.

Meanwhile, Google had a building full of people maintaining the Google kernel (eg, maintaining rejected or unsubmitted patches that were critical for business reasons), and it took many months to do a kernel release.

Re: Linux network performance parameters

#60

Post by dekhn »

pengaru wrote: post #56
klabb3 wrote: post #49

A random thing I ran into with the defaults (Ubuntu Linux): - net.ipv4.tcp_rmem ~ 6MB - net.core.rmem_max ~ 1MB So.. the tcp_rmem value overrides by default, meaning that the TCP receive window for a vanilla TCP socket actually goes up to 6MB if needed (in reality - 3MB because of the halving, but let's ignore that for now since it's a constant). But if I "setsockopt SO_RCVBUF" in a user-space application, I'm actual…

net.ipv4.tcp_rmem max is a limit for the auto-tuning the kernel performs once you do SO_RCVBUF the auto-tuning is out of the picture for that socket, and net.core.rmem_max becomes the max. It's pretty clearly documented @ Documentation/networking/ip-sysctl.rst Edit: downvotes, really? smh

And to add: the kernel autotunes better than you can, so leave that enabled unless you're Vint Cert, Jim Gettys, or Vern Paxton.
Post reply on HN