Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux network performance parameters

github.com

Post reply on HN
Print view View original

81–90 of 117 posts

Re: Linux network performance parameters

#81

Post by zamadatix »

suprjami wrote: post #67
napkin wrote: post #50

Just changing Linux's default congestion control (net.ipv4.tcp_congestion_control) to 'bbr' can make a _huge_ difference in some scenarios, I guess over distances with sporadic packet loss and jitter, and encapsulation. Over the last year, I was troubleshooting issues with the following connection flow: client host reverse proxy host service host On average, I could not get better than 20% theoretical max throughput.…

> BBR Please stop. BBRv1 is broken and should not be used on the open internet. This sort of copy-paste cargo-cult performance tuning (just set a magical value and things will be better) is the exact opposite of what TFA is about. Thankfully Google are upstreaming BBRv3 so this will be over soon.

Yes, BBRv1 has fairness issues when used at scale vs certain other algorithms. No, that doesn't mean people finding it tunes their performance 5x in a particular use case with high latency and some loss should stop talking about how it helps in that scenario. YouTube ran it without the internet burning to the ground so using it in these niche kinds of cases with personal tuning almost certainly results in a net good even though the algorithm wasn't perfect. BBRv3 will make it scale to everyone with better fairness for sure but BBR is still much better behaving for fairness than almost any UDP stream anyways.

Re: Linux network performance parameters

#82

Post by the8472 »

klabb3 wrote: post #49

A random thing I ran into with the defaults (Ubuntu Linux): - net.ipv4.tcp_rmem ~ 6MB - net.core.rmem_max ~ 1MB So.. the tcp_rmem value overrides by default, meaning that the TCP receive window for a vanilla TCP socket actually goes up to 6MB if needed (in reality - 3MB because of the halving, but let's ignore that for now since it's a constant). But if I "setsockopt SO_RCVBUF" in a user-space application, I'm actual…

> (Same applies to SO_SNDBUF/wmem...)

If you want to limit the amount of excess buffered data you can lower TCP_NOTSENT_LOWAT instead, which caps the amount that is buffered beyond what's needed for the BDP.

Re: Linux network performance parameters

#83

Post by fragmede »

suprjami wrote: post #67
napkin wrote: post #50

Just changing Linux's default congestion control (net.ipv4.tcp_congestion_control) to 'bbr' can make a _huge_ difference in some scenarios, I guess over distances with sporadic packet loss and jitter, and encapsulation. Over the last year, I was troubleshooting issues with the following connection flow: client host reverse proxy host service host On average, I could not get better than 20% theoretical max throughput.…

> BBR Please stop. BBRv1 is broken and should not be used on the open internet. This sort of copy-paste cargo-cult performance tuning (just set a magical value and things will be better) is the exact opposite of what TFA is about. Thankfully Google are upstreaming BBRv3 so this will be over soon.

The original cargo cultists built runways on islands to cause supplies to be dropped off. It didn't work. If someone copy and pasted something they don't fully understand off the Internet, but it works, can you really blame them for it, or call them cargo cultists?

Re: Linux network performance parameters

#84

Post by pengaru »

superb_dev wrote: post #77
pengaru wrote: post #71

Earlier quoted context omitted.

[flagged]

Their criticism was accurate and well intentioned. Getting downvoted not for the content but perhaps poor phrasing is perfectly normal. Complaining at all about the votes your internet comment gets is asinine.

It's not asinine to complain that for no good reason a perfectly good technical reference written for the benefit of all readers was being grayed out (at the time) via downvotes. It took a non-zero amount of work to dig up where the setting is documented, and I didn't do it for my own benefit.

This isn't taking it personally like I value HN karma. This is complaining purely because downvotes can make content invisible.

Re: Linux network performance parameters

#86

Post by heybrendan »

drewg123 wrote: post #57
napkin wrote: post #50

Just changing Linux's default congestion control (net.ipv4.tcp_congestion_control) to 'bbr' can make a _huge_ difference in some scenarios, I guess over distances with sporadic packet loss and jitter, and encapsulation. Over the last year, I was troubleshooting issues with the following connection flow: client host reverse proxy host service host On average, I could not get better than 20% theoretical max throughput.…

The difference is that BBR does not use loss as a signal of congestion. Most TCP stacks will cut their send windows in half (or otherwise greatly reduce them) at the first sign of loss. So if you're on a lossy VPN, or sending a huge burst at 1Gb/s on a 10Mb/s VPN uplink, TCP will normally see loss, and back way off. BBR tries to find Bottleneck Bandwidth rate. Eg, the bandwidth of the narrowest or most congested link…

You seem quite knowledgeable in this domain. Have you authored any blog posts to expand on this topic? I would welcome the chance to learn more from you.

Re: Linux network performance parameters

#87

Post by drewg123 »

heybrendan wrote: post #86
drewg123 wrote: post #57

Earlier quoted context omitted.

The difference is that BBR does not use loss as a signal of congestion. Most TCP stacks will cut their send windows in half (or otherwise greatly reduce them) at the first sign of loss. So if you're on a lossy VPN, or sending a huge burst at 1Gb/s on a 10Mb/s VPN uplink, TCP will normally see loss, and back way off. BBR tries to find Bottleneck Bandwidth rate. Eg, the bandwidth of the narrowest or most congested link…

You seem quite knowledgeable in this domain. Have you authored any blog posts to expand on this topic? I would welcome the chance to learn more from you.

I know a little bit about TCP, but I mainly focus on performance. I have several recent talks.. the slides are at: https://people.freebsd.org/~gallatin/talks/

Re: Linux network performance parameters

#88

Post by slug »

superb_dev wrote: post #77
pengaru wrote: post #71

Earlier quoted context omitted.

[flagged]

Their criticism was accurate and well intentioned. Getting downvoted not for the content but perhaps poor phrasing is perfectly normal. Complaining at all about the votes your internet comment gets is asinine.

> Getting downvoted not for the content but perhaps poor phrasing is perfectly normal

IMHO, good and relevant content beats poor phrasing (which again IMHO I didn't witness in the original post), especially since English is not the first language for many people on this board. Downvoting only disincentivizes posting and unfortunately the HN voting system doesn't indicate why, leaving one just to guess.

Re: Linux network performance parameters

#90

Post by klabb3 »

pengaru wrote: post #56
klabb3 wrote: post #49

A random thing I ran into with the defaults (Ubuntu Linux): - net.ipv4.tcp_rmem ~ 6MB - net.core.rmem_max ~ 1MB So.. the tcp_rmem value overrides by default, meaning that the TCP receive window for a vanilla TCP socket actually goes up to 6MB if needed (in reality - 3MB because of the halving, but let's ignore that for now since it's a constant). But if I "setsockopt SO_RCVBUF" in a user-space application, I'm actual…

net.ipv4.tcp_rmem max is a limit for the auto-tuning the kernel performs once you do SO_RCVBUF the auto-tuning is out of the picture for that socket, and net.core.rmem_max becomes the max. It's pretty clearly documented @ Documentation/networking/ip-sysctl.rst Edit: downvotes, really? smh

> once you do SO_RCVBUF the auto-tuning is out of the picture for that socket

Oh I didn’t realize this. That explains the switch in limits. However:

I would have liked to keep auto-tuning, but only change the max buffer size. It’s still weird to me that these are different modes with different limits and whatnot. In my case, I was parallelizing tcp and capping the max size would have been better, and instead varying the number of conns.

I gave up on it. Especially since I need cross platform user-space only, I don’t want to fiddle with these APIs that are all different and unpredictable. I guess it’s for the best anyway, to avoid as much per-platform hacks as possible.

> It's pretty clearly documented @ Documentation/networking/ip-sysctl.rst

I guess I need to step up my doc grepping game, cause it was quite hard to even find this on Google. I ran my own experiments to verify.

> Edit: downvotes, really? smh

Fwiw not me.

Post reply on HN