Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux network performance parameters

github.com

Post reply on HN
Print view View original

41–50 of 117 posts

Re: Linux network performance parameters

#41

Post by dijit »

tptacek wrote: post #38
dijit wrote: post #37

Earlier quoted context omitted.

For one by assuming the work that is done primarily for microkernels/appliances is the absolute limit of userspace networking at Google and that similar work would not go into a hypervisor (hypervisors which are universally treated as a vSwitch in almost all virtual environments the world over). And making that assumption when there are many public examples of Google doing this in other areas such as gVisor and Netst…

If you have information about other userspace networking projects at Google, I'd love to read it, but the Snap paper repeatedly suggests that the userspace networking characteristics of the design are distinctive. Certainly, most networking at Google isn't netstack. Have you done much with netstack? It is many things, but ultra-high-performance isn't one of them.

userspace networking will take different forms depending on the use-case.

Which is one of the arguments of why to do it that way; instead of using general purpose networking.

I haven't the time or inclination to find anything public on this, nor am I interested really in convincing you. Ask a former googler.

Re: Linux network performance parameters

#42

Post by jeffbee »

trustingtrust wrote: post #40
jeffbee wrote: post #21

Earlier quoted context omitted.

i225 is just broken but I get excellent performance from i210. 1gb is hardly challenging on a contemporaneous CPU, and the i210 offers 4 queues. What's your beef with i210?

There are 3 revisions of i225 and Intel essentially got rid of it and launched i226. That one also seems to be problematic [1] . Why is it exponentially harder to make a 2.5gbps NIC when the 1gbps NIC (i210 and i211) has worked well for them. Shouldn't it be trivial to make it 2.5x? They seem to make good 10gbps NICs so I would assume 2.5gbps shouldn't need a 5th try from intel ? [1] - https://shorturl.at/esCNP

The bugs I am aware of are on the PCIe side. i225 will lock up the bus if it attempts to do PTM to support PTP. That's a pretty serious bug. You would think Intel has this nailed since they invented PCIe and PCI for that matter. Apparently not. Maybe they outsourced it.

Re: Linux network performance parameters

#43

Post by sophacles »

circularfoyers wrote: post #25
sophacles wrote: post #20

Earlier quoted context omitted.

Performance parity on which axis? For which use case? Talking generally about "network performance" is approximately as useful as talking generally about "engine performance". Just like it makes no sense to compare a weed-eater engine to a locomotive diesel without talking about use case and desired outcomes, it makes no sense to compare "performance of FreeBSD network stack" and "Linux network stack" without underst…

Can you provide some examples of different contexts where Linux or FreeBSD might be better or worse choices?

Sure:

Linux is a networking swiss army knife (or maybe a dremmel). It can do a lot of stuff reasonably well. It has all sorts of knobs and levers, so you can often configure it to do really weird stuff. I tend to reach for it first to understand the shape of a problem/solution.

BSD is fantastic for a lot of server applications, particularly single tenant high throughput ones like mail servers, dedicated app servers, etc. A great series of case studies have come out of Netflix on this (google for "800Gbps on freebsd netflix" for example - every iteration of that presentation is fantastic and usually discussed here at least once, and Drew G. shows up in comments and answers questions).

It's also pretty nice for firewalling/routing small and medium networks - (opn|pf)sense are both great systems for this built on FreeBSD (apologies for the drama injection this may cause below).

One of the reasons I reach for linux first unless I already know the scope and shape of the problem is that the entire "userland vs kernel" distinction is much blurrier there. Linux allows you to pass some or all traffic to userland at various points in the stack and in various ways, and inject code at the kernel level via ebpf, leading to a lot of hybrid solutions - this is nice in middleboxes where you want some dynamism and control, particularly in multi-tenant networks (and thats the space my work is in, so it's what I know best)

Please bear in mind that these are my opinions and uses/takes on the tools. Just like with programming there's a certain amount of "art" (or maybe "craft") to this, and other folks will have different (but likely just a valid) views - there's a lot of ways to do anything in networking.

Re: Linux network performance parameters

#44

Post by tptacek »

dijit wrote: post #41
tptacek wrote: post #38

Earlier quoted context omitted.

If you have information about other userspace networking projects at Google, I'd love to read it, but the Snap paper repeatedly suggests that the userspace networking characteristics of the design are distinctive. Certainly, most networking at Google isn't netstack. Have you done much with netstack? It is many things, but ultra-high-performance isn't one of them.

userspace networking will take different forms depending on the use-case. Which is one of the arguments of why to do it that way; instead of using general purpose networking. I haven't the time or inclination to find anything public on this, nor am I interested really in convincing you. Ask a former googler.

OK. I did. They said "no, it's not the case that networking at Google is predominately user-mode". (They also said "it depends on what you mean by most"). Do you have more you want me to relay to them? Did you work on this stuff at Google?

Per the Snap thread above: if you're building a router or a load balancer or some other bit of network infrastructure, it's not unlikely that there's userland IP involved. But if you're shipping a normal program on, like, Borg or whatever, it's kernel networking.

Re: Linux network performance parameters

#45

Post by tptacek »

jeffbee wrote: post #39
tptacek wrote: post #29

Earlier quoted context omitted.

Over the course of several years, the architecture underpinning Snap has been used in production for multiple networking applications, including network virtualization for cloud VMs [19], packet-processing for Internet peering [62], scalable load balancing [22], and Pony Express, a reliable transport and communications stack that is our focus for the remainder of this paper. This paper suggests, as I would have expec…

"and Pony Express" is the operative phrase. As the paper states on page 1, "Snap is deployed to over half of our fleet of machines and supports the needs of numerous teams." According to the paper it is not niche.

Isn't Pony Express a ground-up replacement for all of TCP/IP? It doesn't even present a TCP/UDP socket interface.

Re: Linux network performance parameters

#46

Post by dijit »

tptacek wrote: post #44
dijit wrote: post #41

Earlier quoted context omitted.

userspace networking will take different forms depending on the use-case. Which is one of the arguments of why to do it that way; instead of using general purpose networking. I haven't the time or inclination to find anything public on this, nor am I interested really in convincing you. Ask a former googler.

OK. I did. They said "no, it's not the case that networking at Google is predominately user-mode". (They also said "it depends on what you mean by most"). Do you have more you want me to relay to them? Did you work on this stuff at Google? Per the Snap thread above: if you're building a router or a load balancer or some other bit of network infrastructure, it's not unlikely that there's userland IP involved. But if y…

I worked as a Google partner for some specialised projects within AAA online gaming.

I continue in a similar position today and thus my NDA is still in complete effect which limits what I can say if there’s nothing public.

I have not worked for Google, just very closely.

Re: Linux network performance parameters

#47

Post by nolist_policy »

jeffbee wrote: post #39
tptacek wrote: post #29

Earlier quoted context omitted.

Over the course of several years, the architecture underpinning Snap has been used in production for multiple networking applications, including network virtualization for cloud VMs [19], packet-processing for Internet peering [62], scalable load balancing [22], and Pony Express, a reliable transport and communications stack that is our focus for the remainder of this paper. This paper suggests, as I would have expec…

"and Pony Express" is the operative phrase. As the paper states on page 1, "Snap is deployed to over half of our fleet of machines and supports the needs of numerous teams." According to the paper it is not niche.

Makes sense, they're probably using QUIC in lots of products and the kernel can't accelerate that anyways, it would only pass opaque UDP packets to and from the application.

Re: Linux network performance parameters

#48

Post by tptacek »

dijit wrote: post #46
tptacek wrote: post #44

Earlier quoted context omitted.

OK. I did. They said "no, it's not the case that networking at Google is predominately user-mode". (They also said "it depends on what you mean by most"). Do you have more you want me to relay to them? Did you work on this stuff at Google? Per the Snap thread above: if you're building a router or a load balancer or some other bit of network infrastructure, it's not unlikely that there's userland IP involved. But if y…

I worked as a Google partner for some specialised projects within AAA online gaming. I continue in a similar position today and thus my NDA is still in complete effect which limits what I can say if there’s nothing public. I have not worked for Google, just very closely.

Oh. Then, unless a Googler jumps in here and says I'm wrong: no, ordinary applications at Google are not as a rule built on userspace networking. That's not my opinion (though: it was my prior, having done a bunch of userspace networking stuff), it's the result of asking Google people about it.

Maybe it's all changed in the last year! But then: that makes all of this irrelevant to the thread, about FreeBSD vs. Linux network stack performance.

Re: Linux network performance parameters

#49

Post by klabb3 »

A random thing I ran into with the defaults (Ubuntu Linux):

- net.ipv4.tcp_rmem ~ 6MB

- net.core.rmem_max ~ 1MB

So.. the tcp_rmem value overrides by default, meaning that the TCP receive window for a vanilla TCP socket actually goes up to 6MB if needed (in reality - 3MB because of the halving, but let's ignore that for now since it's a constant).

But if I "setsockopt SO_RCVBUF" in a user-space application, I'm actually capped at a maximum 1MB, even though I already have 6MB. If I try to reduce it from 6MB to e.g. 4MB, it will result in 1MB. This seems very strange. (Perhaps I'm holding it wrong?)

(Same applies to SO_SNDBUF/wmem...)

To me, it seems like Linux is confused about the precedence order of these options. Why not have core.rmem_max be larger and the authoritative directive? Is there some historical reason for this?

Re: Linux network performance parameters

#50

Post by napkin »

Just changing Linux's default congestion control (net.ipv4.tcp_congestion_control) to 'bbr' can make a _huge_ difference in some scenarios, I guess over distances with sporadic packet loss and jitter, and encapsulation.

Over the last year, I was troubleshooting issues with the following connection flow:

client host reverse proxy host service host

On average, I could not get better than 20% theoretical max throughput. Also, connections tended to slow to a crawl over time. I had hacky solutions like forcing connections to close frequently. Finally switching congestion control to 'bbr' gives close to theoretical max throughput and reliable connections.

I don't really understand enough about TCP to understand why it works. The change needed to be made on both sides of Wireguard.

Post reply on HN