Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux network performance parameters

github.com

Post reply on HN
Print view View original

61–70 of 117 posts

Re: Linux network performance parameters

#61

Post by tptacek »

drewg123 wrote: post #59
tptacek wrote: post #30

Earlier quoted context omitted.

Do you know for a fact that Google primarily uses userland networking, or does that just seem accurate to you?

Google makes heavy use of userspace networking. I was there roughly a decade ago. At least at that time, a major factor is the choice of userspace over kernel networking was time to deployment. Services like the ones described above were built on the monorepo, and could be deployed in seconds at the touch of a button. Meanwhile, Google had a building full of people maintaining the Google kernel (eg, maintaining rejec…

Yes. I don't think anyone is disputing that Google does significant userspace networking things. But the premise of this thread is that "ordinary" (ie: non-network-infrastructure --- SDN, load balancer, routing) applications, things that would normally just get BSD sockets, are based on userspace networking. That seems not to be the case.

Re: Linux network performance parameters

#63

Post by LukeShu »

pengaru wrote: post #56
klabb3 wrote: post #49

A random thing I ran into with the defaults (Ubuntu Linux): - net.ipv4.tcp_rmem ~ 6MB - net.core.rmem_max ~ 1MB So.. the tcp_rmem value overrides by default, meaning that the TCP receive window for a vanilla TCP socket actually goes up to 6MB if needed (in reality - 3MB because of the halving, but let's ignore that for now since it's a constant). But if I "setsockopt SO_RCVBUF" in a user-space application, I'm actual…

net.ipv4.tcp_rmem max is a limit for the auto-tuning the kernel performs once you do SO_RCVBUF the auto-tuning is out of the picture for that socket, and net.core.rmem_max becomes the max. It's pretty clearly documented @ Documentation/networking/ip-sysctl.rst Edit: downvotes, really? smh

1. While your context about auto-tuning is accurate and valuable, it doesn't really address the fundamental strangeness that the parent post is commenting about: It's still strange that it can auto-tune to a higher value than you can manually tune it to.

2. It's always valuable to provide further references, but I'd guess that down-voters found the "It's pretty clearly documented" phrasing a little condescending? Perhaps "See the docs at [] for more information."?

3. "Please don't comment about the voting on comments. It never does any good, and it makes boring reading."

Re: Linux network performance parameters

#64

Post by toast0 »

Thaxll wrote: post #5
mikece wrote: post #4

[flagged]

This is kind of an urban legend, do you think the multi millions servers from Google, Amazon etc... have those performance issues?

I've been at two large FreeBSD users that switched to Linux, and the reason was never performance.

Yahoo switched to Linux because having a single server OS is nicer than having two, acquisitions were nearly all running Linux, and Linux was at least good enough (although having all of Overture crash at the same time because of a leap second bug in the Linux kernel, twice, wasn't great), and maybe something about easier to hire for kernel engineers.

WhatsApp switched to Linux because the server team was mostly ex-Yahoo and had seen how much strife running a different OS than your acquirer in their datacenters causes. There was enough non-negotiable strife from all of the other tech and philosophy differences, that accepting the acquirer kernel that is at least good enough was worthwhile.

I'm not saying FreeBSD or Linux has better performance. I like the reputation FreeBSD has for network performance, and it certainly has good performance, but is it better than Linux? I don't know, I never ran apples to apples comparisons, because whenever I was involved in a switch, the hardware was going to be very different, and the changeover was a policy decision rather than a technical one.

Personally, I run FreeBSD when I can, and Linux when I have to. FreeBSD's development model and smaller team lead to less churn, and I value stability and consistency. Other people have different values, and that's fine too.

Re: Linux network performance parameters

#65

Post by ghotli »

trustingtrust wrote: post #40
jeffbee wrote: post #21

Earlier quoted context omitted.

i225 is just broken but I get excellent performance from i210. 1gb is hardly challenging on a contemporaneous CPU, and the i210 offers 4 queues. What's your beef with i210?

There are 3 revisions of i225 and Intel essentially got rid of it and launched i226. That one also seems to be problematic [1] . Why is it exponentially harder to make a 2.5gbps NIC when the 1gbps NIC (i210 and i211) has worked well for them. Shouldn't it be trivial to make it 2.5x? They seem to make good 10gbps NICs so I would assume 2.5gbps shouldn't need a 5th try from intel ? [1] - https://shorturl.at/esCNP

I'm interested in clicking your link just not through a shortener. Perhaps just my own bias but figured I'd surface that reaction here. Much more useful to see the domain I'll end up on.

Re: Linux network performance parameters

#66

Post by dijit »

tptacek wrote: post #48
dijit wrote: post #46

Earlier quoted context omitted.

I worked as a Google partner for some specialised projects within AAA online gaming. I continue in a similar position today and thus my NDA is still in complete effect which limits what I can say if there’s nothing public. I have not worked for Google, just very closely.

Oh. Then, unless a Googler jumps in here and says I'm wrong: no, ordinary applications at Google are not as a rule built on userspace networking. That's not my opinion (though: it was my prior, having done a bunch of userspace networking stuff), it's the result of asking Google people about it. Maybe it's all changed in the last year! But then: that makes all of this irrelevant to the thread, about FreeBSD vs. Linux…

Based on this I understand why you're talking like this: I think you have made an assumption/interpretation here and argued the assumption because nobody here (I believe) has claimed that Google only uses user-space networking; merely that google makes use of user-space networking where it's "appropriate" (IE; when FreeBSD would have had an advantage). Which is backed up by basically everything in this thread.

Which is why I said you "probably read it wrong".

Google is much happier to throw hardware at the problem in most cases, only when it really matters and they would have had to rearchitect the kernel to improve a situation any further do they break out the user-space networking.

The point I was driving at was that it's more common than you think.

Your base assertion that it's ubiquitous is very obviously false because Chromebooks are pretty common inside google offices and those are running stock chromeOS (except in the offices that are developing chromeOS)

Re: Linux network performance parameters

#67

Post by suprjami »

napkin wrote: post #50

Just changing Linux's default congestion control (net.ipv4.tcp_congestion_control) to 'bbr' can make a _huge_ difference in some scenarios, I guess over distances with sporadic packet loss and jitter, and encapsulation. Over the last year, I was troubleshooting issues with the following connection flow: client host reverse proxy host service host On average, I could not get better than 20% theoretical max throughput.…

> BBR

Please stop. BBRv1 is broken and should not be used on the open internet.

This sort of copy-paste cargo-cult performance tuning (just set a magical value and things will be better) is the exact opposite of what TFA is about.

Thankfully Google are upstreaming BBRv3 so this will be over soon.

Re: Linux network performance parameters

#68

Post by tptacek »

dijit wrote: post #66
tptacek wrote: post #48

Earlier quoted context omitted.

Oh. Then, unless a Googler jumps in here and says I'm wrong: no, ordinary applications at Google are not as a rule built on userspace networking. That's not my opinion (though: it was my prior, having done a bunch of userspace networking stuff), it's the result of asking Google people about it. Maybe it's all changed in the last year! But then: that makes all of this irrelevant to the thread, about FreeBSD vs. Linux…

Based on this I understand why you're talking like this: I think you have made an assumption/interpretation here and argued the assumption because nobody here (I believe) has claimed that Google only uses user-space networking; merely that google makes use of user-space networking where it's "appropriate" (IE; when FreeBSD would have had an advantage). Which is backed up by basically everything in this thread. Which…

Let me make my point clearly: Google depends on the Linux kernel stack as much as the top-of-the-thread comment suggests that they did, and the things they're doing in user-mode, they would also be doing in user-mode on FreeBSD.

That's all I'm here to say.

As these things go, in the course of making your argument, you made a falsifiable and, I believe, flatly incorrect claim:

Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking

At least in Google's case, this isn't true. People doing custom network stack stuff totally do bypass the kernel stack (sometimes with usermode stacks, and sometimes with eBPF, and sometimes with offload). But the way you phrased this, you implied pretty directly that networking writ large was usermode at Google, and while I entertained the possibility that this might be true, when I investigated, it wasn't (unsurprisingly, given how annoying user mode networking is to interact directly with, as opposed to in middlebox applications).

Re: Linux network performance parameters

#69

Post by suprjami »

jeffbee wrote: post #13
corbet wrote: post #12

Earlier quoted context omitted.

That's funny ... the "big guys" are some of the biggest contributors to the Linux network stack, almost as if they were actually using it and cared about how well it works.

History has shown that tons of Linux networking scalability and performance contributions have been rejected by the gatekeepers/maintainers. The upstream kernel remains unsuitable for datacenter use, and all the major operators bypass or patch it.

Look who you're arguing with.

Re: Linux network performance parameters

#70

Post by suprjami »

doctorpangloss wrote: post #16

Does performance tuning for Wi-Fi adapters matter? On desktops, other than disabling features, can anything fix the problems with i210 and i225 ethernet? Those seem to be the two most common NICs nowadays. I don't really understand why common networking hardware and drivers are so flawed. There is a lot of attention paid to RISC-V. How about start with a fully open and correct NIC? They'll shove it in there if it's c…

I have a motherboard with an i225 onboard.

I bought a PCIe I350. That's solved the problem.

Post reply on HN