Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's always TCP_NODELAY

brooker.co.za

Post reply on HN
Print view View original

181–190 of 276 posts

Re: It's always TCP_NODELAY

#181

Post by pzs »

nh2 wrote: post #134
ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

Same here. I have a hobby that on any RPC framework I encounter, I file a Github issue "did you think of TCP_NODELAY or can this framework do only 20 calls per second?". So far, it's found a bug every single time. Some examples: https://cloud-haskell.atlassian.net/browse/DP-108 or https://github.com/agentm/curryer/issues/3 I disagree on the "not a good / bad option" though. It's a kernel-side heuristic for "magically…

"As the article states, no sensible application does 1-byte network write() syscalls." - the problem that this flag was meant to solve was that when a user was typing at a remote terminal, which used to be a pretty common use case in the 80's (think telnet), there was one byte available to send at a time over a network with a bandwidth (and latency) severely limited compared to today's networks. The user was happy to see that the typed character arrived to the other side. This problem is no longer significant, and the world has changed so that this flag has become a common issue in many current use cases.

Was terminal software poorly written? I don't feel comfortable to make such judgement. It was designed for a constrained environment with different priorities.

Anyway, I agree with the rest of your comment.

Re: It's always TCP_NODELAY

#183

Post by userbinator »

EvanAnderson wrote: post #108
evanelias wrote: post #31

John Nagle has posted insightful comments about the historical background for this many times, for example https://news.ycombinator.com/item?id=9048947 referenced in the article. He's a prolific HN commenter (#11 on the leaderboard) so it can be hard to find everything, but some more comments searchable via https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... or https://hn.algolia.com/?dateRange=all&page=0…

I love it when Nagle's algorithm comes up on HN. Inevitably someone, not knowing "Animats" is John Nagle, responds a comment from Animats with a "knowing better" tone. >smile (I also really like Animats' comments, too.)

I always check if the man himself makes an appearance every time I see that. He has posted a few comments in here already.

Re: It's always TCP_NODELAY

#184

Post by hi-v-rocknroll »

Apropos repost from 2015:

> That still irks me. The real problem is not tinygram prevention. It's ACK delays, and that stupid fixed timer. They both went into TCP around the same time, but independently. I did tinygram prevention (the Nagle algorithm) and Berkeley did delayed ACKs, both in the early 1980s. The combination of the two is awful. Unfortunately by the time I found about delayed ACKs, I had changed jobs, was out of networking, and doing a product for Autodesk on non-networked PCs.

> Delayed ACKs are a win only in certain circumstances - mostly character echo for Telnet. (When Berkeley installed delayed ACKs, they were doing a lot of Telnet from terminal concentrators in student terminal rooms to host VAX machines doing the work. For that particular situation, it made sense.) The delayed ACK timer is scaled to expected human response time. A delayed ACK is a bet that the other end will reply to what you just sent almost immediately. Except for some RPC protocols, this is unlikely. So the ACK delay mechanism loses the bet, over and over, delaying the ACK, waiting for a packet on which the ACK can be piggybacked, not getting it, and then sending the ACK, delayed. There's nothing in TCP to automatically turn this off. However, Linux (and I think Windows) now have a TCP_QUICKACK socket option. Turn that on unless you have a very unusual application.

> Turning on TCP_NODELAY has similar effects, but can make throughput worse for small writes. If you write a loop which sends just a few bytes (worst case, one byte) to a socket with "write()", and the Nagle algorithm is disabled with TCP_NODELAY, each write becomes one IP packet. This increases traffic by a factor of 40, with IP and TCP headers for each payload. Tinygram prevention won't let you send a second packet if you have one in flight, unless you have enough data to fill the maximum sized packet. It accumulates bytes for one round trip time, then sends everything in the queue. That's almost always what you > want. If you have TCP_NODELAY set, you need to be much more aware of buffering and flushing issues.

> None of this matters for bulk one-way transfers, which is most HTTP today. (I've never looked at the impact of this on the SSL handshake, where it might matter.)

> Short version: set TCP_QUICKACK. If you find a case where that makes things worse, let me know.

> John Nagle

(2015)

https://news.ycombinator.com/item?id=10608356

---

Support platform survey:

TCP_QUICKACK: Linux (must set again every every recv())

TCP_NODELAY: Linux, Apple, Windows, Solaris, FreeBSD, OpenBSD, and NetBSD

References:

https://www.man7.org/linux/man-pages/man7/tcp.7.html

https://opensource.apple.com/source/xnu/xnu-1504.9.17/bsd/ne...

https://learn.microsoft.com/en-us/windows/win32/winsock/ippr...

https://docs.oracle.com/cd/E88353_01/html/E37851/esc-tcp-4p....

https://man.freebsd.org/cgi/man.cgi?query=tcp

https://man.openbsd.org/tcp

https://man.netbsd.org/NetBSD-8.0/tcp.4

Re: It's always TCP_NODELAY

#185

Post by School-Cotton »

dataflow wrote: post #177
Spivak wrote: post #175

Earlier quoted context omitted.

I think it's a distinction without a difference in this case. You can't know if the reason your water stopped is because the water is shut off, the pipe broke, or it's just slow. When all you have to go on is "I stopped getting packets" the best you can do is give up after a bit. TCP keepsalives do kinda suck and are full of interesting choices that don't seem to have passed the test of time. But they are there and i…

There's a crucial difference in fact, which is that the peer you're defining connectedness to is a single well-defined peer that is directly connected to you, which "The Network" is not. As for the analogy, uh, this ain't water. Monitor the line voltage or the fiber brightness or something, it'll tell you very quickly if the other endpoint is disconnected. It's up to the physical layer to provide a mechanism to detec…

Well, isn't that already how it works? If I physically unplug my ethernet cable, won't TCP-related syscalls start failing immediately?

Re: It's always TCP_NODELAY

#186

Post by SoftTalker »

pzs wrote: post #181
nh2 wrote: post #134

Earlier quoted context omitted.

Same here. I have a hobby that on any RPC framework I encounter, I file a Github issue "did you think of TCP_NODELAY or can this framework do only 20 calls per second?". So far, it's found a bug every single time. Some examples: https://cloud-haskell.atlassian.net/browse/DP-108 or https://github.com/agentm/curryer/issues/3 I disagree on the "not a good / bad option" though. It's a kernel-side heuristic for "magically…

"As the article states, no sensible application does 1-byte network write() syscalls." - the problem that this flag was meant to solve was that when a user was typing at a remote terminal, which used to be a pretty common use case in the 80's (think telnet), there was one byte available to send at a time over a network with a bandwidth (and latency) severely limited compared to today's networks. The user was happy to…

> when a user was typing at a remote terminal, which used to be a pretty common use case in the 80's

Still is for some. I’m probably working in a terminal on an ssh connection to a remote system for 80% of my work day.

Re: It's always TCP_NODELAY

#187

Post by hi-v-rocknroll »

mort96 wrote: post #71
koverstreet wrote: post #38

Earlier quoted context omitted.

Like TCP keepalives?

If the feature already technically exists in TCP, it's either broken or disabled by default, which is pretty much the same as not having it.

You're conflating all optional TCP features of all operating systems, network devices, and RFCs together. This lack of nuance fails to appreciate that different applications have different needs for how they use TCP: ( server | client ) x ( one way | chatty bidirectional | idle tinygram | mixed ). If a feature needs to be used on a particular connection, then use it. ;)

Re: It's always TCP_NODELAY

#188

Post by dgoldstein0 »

SoftTalker wrote: post #186
pzs wrote: post #181

Earlier quoted context omitted.

"As the article states, no sensible application does 1-byte network write() syscalls." - the problem that this flag was meant to solve was that when a user was typing at a remote terminal, which used to be a pretty common use case in the 80's (think telnet), there was one byte available to send at a time over a network with a bandwidth (and latency) severely limited compared to today's networks. The user was happy to…

> when a user was typing at a remote terminal, which used to be a pretty common use case in the 80's Still is for some. I’m probably working in a terminal on an ssh connection to a remote system for 80% of my work day.

sure, but we do so with much better networks than in the 80s. The extra overhead is not going to matter when even a bad network nowadays is measured in megabits per second per user. The 80s had no such luxury.

Re: It's always TCP_NODELAY

#189

Post by hi-v-rocknroll »

landswipe wrote: post #105

Use UDP ;)

Too many applications end up reinventing TCP or SCTP in user-space. Also, network-level QoS applied to unrecognized UDP protocols typically means it gets throttled before TCP. Use UDP when nothing else will work, when the use-case doesn't need a persistent connection, and when no other messaging or transport library is suitable.

Re: It's always TCP_NODELAY

#190

Post by dataflow »

School-Cotton wrote: post #185
dataflow wrote: post #177

Earlier quoted context omitted.

There's a crucial difference in fact, which is that the peer you're defining connectedness to is a single well-defined peer that is directly connected to you, which "The Network" is not. As for the analogy, uh, this ain't water. Monitor the line voltage or the fiber brightness or something, it'll tell you very quickly if the other endpoint is disconnected. It's up to the physical layer to provide a mechanism to detec…

Well, isn't that already how it works? If I physically unplug my ethernet cable, won't TCP-related syscalls start failing immediately?

Probably, but I don't know how the physical layers work underneath. But regardless, it's trivial to just monitor something constantly to ensure the connection is still there, you just need the hardware and protocol support.
Post reply on HN