Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux network performance parameters

github.com

Post reply on HN
Print view View original

71–80 of 117 posts

Re: Linux network performance parameters

#71

Post by pengaru »

LukeShu wrote: post #63
pengaru wrote: post #56

Earlier quoted context omitted.

net.ipv4.tcp_rmem max is a limit for the auto-tuning the kernel performs once you do SO_RCVBUF the auto-tuning is out of the picture for that socket, and net.core.rmem_max becomes the max. It's pretty clearly documented @ Documentation/networking/ip-sysctl.rst Edit: downvotes, really? smh

1. While your context about auto-tuning is accurate and valuable, it doesn't really address the fundamental strangeness that the parent post is commenting about: It's still strange that it can auto-tune to a higher value than you can manually tune it to. 2. It's always valuable to provide further references, but I'd guess that down-voters found the "It's pretty clearly documented" phrasing a little condescending? Per…

[flagged]

Re: Linux network performance parameters

#72

Post by suprjami »

gjulianm wrote: post #3

This is great, not just the parameters themselves but all the steps that a packet follows from the point it enters the NIC until it gets to userspace. Just one thing to add regarding network performance: if you're working in a system with multiple CPUs (which is usually the case in big servers), check NUMA allocation. Sometimes the network card will be in one CPU while the application is executing on a different one,…

Packagecloud have a great article series which goes into much more detail and with code study. If you really want to learn the network send and receive, these are the articles to read:

https://blog.packagecloud.io/monitoring-tuning-linux-network...

https://blog.packagecloud.io/monitoring-tuning-linux-network...

https://packagecloud.io/blog/illustrated-guide-monitoring-tu...

Re: Linux network performance parameters

#74

Post by dijit »

tptacek wrote: post #68
dijit wrote: post #66

Earlier quoted context omitted.

Based on this I understand why you're talking like this: I think you have made an assumption/interpretation here and argued the assumption because nobody here (I believe) has claimed that Google only uses user-space networking; merely that google makes use of user-space networking where it's "appropriate" (IE; when FreeBSD would have had an advantage). Which is backed up by basically everything in this thread. Which…

Let me make my point clearly: Google depends on the Linux kernel stack as much as the top-of-the-thread comment suggests that they did, and the things they're doing in user-mode, they would also be doing in user-mode on FreeBSD. That's all I'm here to say. As these things go, in the course of making your argument, you made a falsifiable and, I believe, flatly incorrect claim: Most of the larger tech companies these d…

Ok, this is a very hostile and could not possibly be considered a charitable interpretation of what I said, in fact I'd say it borders on trying to pick an argument where there isn't one. I did not "flat out lie" and I detest the insinuation. I expect better of you honestly.

To answer: "Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking"

You could read "almost completely" as in "almost across the whole company" which would be a weird way to read it, but you seem to have read it this way.

I intended it to mean: when they bypass the kernel; it is a near complete bypass.

Since there actually is still a network connection going through the kernel (the host itself will still be connected of course), which is of course the inverse of what you seem to have taken away; in that user-mode networking is used even less than entirely even on a single node.

edit: In fact, I stated multiple times in my post: "Google is mostly happy to just throw hardware at this" which you seemed to just.. ignore? Google are absolutely happy to throw hardware at issues until they can't anymore or the gains are too enormous to avoid. I thought I was extremely clear about that.

Your point about user-land networking in FreeBSD is just a nonsense one to make -- and not the point we were discussing anyway, like suggesting "if it did rain beer, would we all get drunk?" it's completely hypothetical and not based in any subjective reality or objective truth. You have absolutely no way of knowing if FreeBSD could do those things, the statement that the architecture permits is is shown somewhat in FreeBSD's use in Netflix, which has been commented elsewhere in this thread to achieve close to 0.8TiB of data transfer, but knowing what google would have done with freebsd would require seeing into alternative realities.

I don't know where you work, but as far as I know: nobody has managed to perfect that technology yet.

Re: Linux network performance parameters

#75

Post by suprjami »

doctorpangloss wrote: post #54
jeffbee wrote: post #21

Earlier quoted context omitted.

i225 is just broken but I get excellent performance from i210. 1gb is hardly challenging on a contemporaneous CPU, and the i210 offers 4 queues. What's your beef with i210?

There are a lot of problems with the i210. Here’s a sample: https://www.google.com/search?q=i210+proxmox+e1000e+disable Most people don’t really use their NICs “all the time” “with many hosts.” The i210 in particular will hang after a few months of e.g. etcd cluster traffic on 9th and 10th gen Intel, which is common for SFFPCs. On Windows, the ndis driver works a lot better. Many disconnects in similar traffic load a…

I like your better I/O idea.

Many systems which cost more than a good car are still coming with the Broadcom 5719 (tg3) from 1999. They have a single transmit queue and the driver is full of workarounds. It's a complete joke these are still supplied today.

SFP would be great but I'd settle for an onboard NIC chipset which was made in the last 10 years.

Re: Linux network performance parameters

#76

Post by i_am_a_peasant »

pengaru wrote: post #71
LukeShu wrote: post #63

Earlier quoted context omitted.

1. While your context about auto-tuning is accurate and valuable, it doesn't really address the fundamental strangeness that the parent post is commenting about: It's still strange that it can auto-tune to a higher value than you can manually tune it to. 2. It's always valuable to provide further references, but I'd guess that down-voters found the "It's pretty clearly documented" phrasing a little condescending? Per…

[flagged]

for what it's worth i found the info you provided good to know

Re: Linux network performance parameters

#77

Post by superb_dev »

pengaru wrote: post #71
LukeShu wrote: post #63

Earlier quoted context omitted.

1. While your context about auto-tuning is accurate and valuable, it doesn't really address the fundamental strangeness that the parent post is commenting about: It's still strange that it can auto-tune to a higher value than you can manually tune it to. 2. It's always valuable to provide further references, but I'd guess that down-voters found the "It's pretty clearly documented" phrasing a little condescending? Per…

[flagged]

Their criticism was accurate and well intentioned. Getting downvoted not for the content but perhaps poor phrasing is perfectly normal. Complaining at all about the votes your internet comment gets is asinine.

Re: Linux network performance parameters

#78

Post by tptacek »

dijit wrote: post #74
tptacek wrote: post #68

Earlier quoted context omitted.

Let me make my point clearly: Google depends on the Linux kernel stack as much as the top-of-the-thread comment suggests that they did, and the things they're doing in user-mode, they would also be doing in user-mode on FreeBSD. That's all I'm here to say. As these things go, in the course of making your argument, you made a falsifiable and, I believe, flatly incorrect claim: Most of the larger tech companies these d…

Ok, this is a very hostile and could not possibly be considered a charitable interpretation of what I said, in fact I'd say it borders on trying to pick an argument where there isn't one. I did not "flat out lie" and I detest the insinuation. I expect better of you honestly. To answer: "Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking" Y…

That is exactly how I took your comment.

This is a weird cursed thread that started out with a pretty silly† claim about FreeBSD vs. Linux network stack performance (neither of us started it; it's a standard platform war argument). Someone made a comment that the hyperscalers all depend on the Linux kernel stack, to a far greater degree than they do on FreeBSD. That's a true statement; at this point, you and I have both agreed on it.

When that point was pressed earlier, you and another comment brought up kernel bypass (usermode networking, specifically) as a way of dismissing hyperscaler Linux dependencies. But that's not really a valid argument. Hyperscalers do kernel bypass stuff! Of course they do! But they're doing it for the things that you'd formerly have bought dedicated network equipment for, and in no case are they doing it in a situation where deploying FreeBSD and using the FreeBSD stack would be a valid alternative.

The disconnect between us is that I'm still talking about the subject that the thread was originally about --- whether Google using the Linux stack is a valid point backing up its fitness for purpose. I think it pretty clearly is a valid point. I think the usermode networking stuff is interesting, but is a sideshow.

† "Silly" because these kinds of claims never, ever get resolved, and just bring out each side's cheering section, not because I have low opinions of FreeBSD's kernel stack --- I came up on that stack and find it easier to follow than Linux's, though I rather doubt the claim that it has a decisive performance advantage.

Re: Linux network performance parameters

#79

Post by dijit »

tptacek wrote: post #78
dijit wrote: post #74

Earlier quoted context omitted.

Ok, this is a very hostile and could not possibly be considered a charitable interpretation of what I said, in fact I'd say it borders on trying to pick an argument where there isn't one. I did not "flat out lie" and I detest the insinuation. I expect better of you honestly. To answer: "Most of the larger tech companies these days are using userland networking and bypass the kernel almost completely for networking" Y…

That is exactly how I took your comment. This is a weird cursed thread that started out with a pretty silly† claim about FreeBSD vs. Linux network stack performance (neither of us started it; it's a standard platform war argument). Someone made a comment that the hyperscalers all depend on the Linux kernel stack, to a far greater degree than they do on FreeBSD. That's a true statement; at this point, you and I have b…

Thank you for clarifying, and I apologise as I also became quite hostile.

I agree that we agree on many points; but I think where we diverge (and perhaps fundamentally) is in the base assumption that "because google uses it, it must be the best, because even if it wasn't google would make it so" (at least, this is my interpretation of GPs comment).

I have little doubt that the low hanging... mid hanging and perhaps even most of the high-hanging fruit has been well and truly plucked when it comes to linux throughput at the behest of the large tech companies; because a few percent improvement translates a lot at their scale. -- However I am reminded of an allegory given in a talk (that I can't find) regarding bowling.

In the talk the speaker mentions how they got "really good" at bowling with completely the wrong technique; but it worked for them, up to a point in which they could not improve no matter what they did. They had to go back to the basics and learn proper technique and become much worse before they were able to overtake their previous scores with the bad technique.... but after that point there were further improvements to be had.

My argument that this is the case is merely: doing an architectural re-write of the linux kernel to be more scalable in the way FreeBSD's is would be very punishing for too many people, and additionally that the economics are not favourable when, if you do get to a point where you cannot scale due to the kernel, you could just break out into userland. -- then simply suffer the adequate but not insane performance everywhere else where it's not needed anyway.

So, to summarise my points:

* Because a big company uses something does not mean it is perfect in all areas

* That Linux has a lot of attention on it does not mean necessarily that it has the most potential: though I don't doubt that the majority of it's potential has been reached.

* Diminishing returns means once it's "good enough" people will try to get performance elsewhere if they need it.

* Rewriting the network stack in Linux completely would likely be harmful to many and subtly so, I haven't seen people moving towards this idea either, this feels like it could be political as well as technical.

* Hyperscalers will often trade convenience over performance: regardless, CPU time is much cheaper for them than it is for us.

Re: Linux network performance parameters

#80

Post by tptacek »

dijit wrote: post #79
tptacek wrote: post #78

Earlier quoted context omitted.

That is exactly how I took your comment. This is a weird cursed thread that started out with a pretty silly† claim about FreeBSD vs. Linux network stack performance (neither of us started it; it's a standard platform war argument). Someone made a comment that the hyperscalers all depend on the Linux kernel stack, to a far greater degree than they do on FreeBSD. That's a true statement; at this point, you and I have b…

Thank you for clarifying, and I apologise as I also became quite hostile. I agree that we agree on many points; but I think where we diverge (and perhaps fundamentally) is in the base assumption that "because google uses it, it must be the best, because even if it wasn't google would make it so" (at least, this is my interpretation of GPs comment). I have little doubt that the low hanging... mid hanging and perhaps e…

I think it's totally legit to say that hyperscaler Linux adoption isn't dispositive of the Linux's stack's performance advantage over FreeBSD. I basically think of this FreeBSD vs. Linux stack debate as unknowable (it probably isn't, but it's liberating to decide I'm not going to resolve it to anyone's satisfaction). So I'm not here to say "Google uses Linux ergo it's faster than FreeBSD"; that adoption is a useful observation, but that's all it is.
Post reply on HN