Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's always TCP_NODELAY

brooker.co.za

Post reply on HN
Print view View original

191–200 of 276 posts

Re: It's always TCP_NODELAY

#191

Post by PaulDavisThe1st »

bjourne wrote: post #122
temac wrote: post #8

Earlier quoted context omitted.

Fix the apps. Nobody expect magical perf if you do that when writing to files, even though the OS also has its own buffers. There is no reason to expect otherwise when writing to a socket and actually nagle already doesn't save you from syscall overhead.

> Fix the apps. Nobody expect magical perf if you do that when writing to files, We write to files line-by-line or even character-by-character and expect the library or OS to "magically" buffer it into fast file writes. Same with memory. We expect multiple small mallocs to be smartly coalesced by the platform.

If you expect a POSIX-y OS to buffer write(2) calls, you're sadly misguided. Whether or not that happens depends on nature of the device file you're writing to.

OTOH, if you're using fwrite(3), as you likely should be actual file I/O, then your expectation is entirely reasonable.

Similarly with memory. If you expect brk(2) to handle multiple small allocations "sensibly" you're going to be disappointed. If you use malloc(3) then your expectation is entirely reasonable.

Re: It's always TCP_NODELAY

#192

Post by lanstin »

unconed wrote: post #106
jcgrillo wrote: post #65

Earlier quoted context omitted.

Configuration changes are one of those areas where having some kind of "are you sure? (y/n)" check can really pay off. It wouldn't have helped in this case, because there wasn't really any change management process to speak of, but we haven't fully learned the lesson yet.

Confirmations are mostly useless unless you explicitly spell out the implications of the change. They are also inferior to being able to undo changes. That's a lesson many don't know.

Your time from commit to live is proportional to your rollback to a known good state. Maybe to a power of the rollback time.

Re: It's always TCP_NODELAY

#193

Post by underdeserver »

SoftTalker wrote: post #186
pzs wrote: post #181

Earlier quoted context omitted.

"As the article states, no sensible application does 1-byte network write() syscalls." - the problem that this flag was meant to solve was that when a user was typing at a remote terminal, which used to be a pretty common use case in the 80's (think telnet), there was one byte available to send at a time over a network with a bandwidth (and latency) severely limited compared to today's networks. The user was happy to…

> when a user was typing at a remote terminal, which used to be a pretty common use case in the 80's Still is for some. I’m probably working in a terminal on an ssh connection to a remote system for 80% of my work day.

If you're working on a distributed system, most of the traffic is not going to be your SSH session though.

Re: It's always TCP_NODELAY

#194

Post by lanstin »

caf wrote: post #168
gizmo686 wrote: post #157

Earlier quoted context omitted.

My guess would be that the server assumes that every call to recv() terminates on a message boundary. With TCP_NODELAY and small messages, this works out fine. Every message is contained in a single packet, and the userspace buffer being read into is large enough to contain it. As such, whenever the kernel has any data to give to userspace, it has an integer number of messages to give. Nothing requires the kernel to…

Otherwise known as the "TCP is a stream-based abstraction, not a packet-based abstraction" bug. A related one is failing to process the second of two complete commands that happen to arrive in the same recv() call.

I find these bugs to be a sign that the app is not using a good wrapper but just mostly gets lucky that the packet isn’t split randomly on the way.

Re: It's always TCP_NODELAY

#195

Post by pipe01 »

Animats wrote: post #80
evanelias wrote: post #31

John Nagle has posted insightful comments about the historical background for this many times, for example https://news.ycombinator.com/item?id=9048947 referenced in the article. He's a prolific HN commenter (#11 on the leaderboard) so it can be hard to find everything, but some more comments searchable via https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... or https://hn.algolia.com/?dateRange=all&page=0…

The sending pattern matters. Send/Receive/Send/Receive won't trigger the problem, because the request will go out immediately and the reply will provide an ACK and allow another request. Bulk transfers won't cause the problem, because if you fill the outgoing block size, there's no delay. But Send/Send/Receive will. This comes up a lot in game systems, where most of the traffic is small events going one way.

I would imagine that games that require exotic sending patterns would use UDP, giving them more control over the protocol

Re: It's always TCP_NODELAY

#196

Post by hgomersall »

nh2 wrote: post #134
ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

Same here. I have a hobby that on any RPC framework I encounter, I file a Github issue "did you think of TCP_NODELAY or can this framework do only 20 calls per second?". So far, it's found a bug every single time. Some examples: https://cloud-haskell.atlassian.net/browse/DP-108 or https://github.com/agentm/curryer/issues/3 I disagree on the "not a good / bad option" though. It's a kernel-side heuristic for "magically…

Would one not also get clobbered by all the sys calls for doing many small packets? It feels like coalescing in userspace is a much better strategy all round if that's desired, but I'm not super experienced.

Re: It's always TCP_NODELAY

#197

Post by supriyo-biswas »

klabb3 wrote: post #133
niutech wrote: post #102

Earlier quoted context omitted.

Shouldn't QUIC ( https://en.wikipedia.org/wiki/QUIC ) solve the TCP issues like latency?

As someone who needed high throughput and looked to QUIC because of control of buffers, I recommend against it at this time. It’s got tons of performance problems depending on impl and the API is different. I don’t think QUIC is bad, or even overengineered, really. It delivers useful features, in theory, that are quite well designed for the modern web centric world. Instead I got a much larger appreciation for TCP, a…

Was this ever implemented though? I found [1] but it was frozen due to age and was never worked on, it seems.

(Edit: doing some more reading, it seems TCP_NODELAY was always the default in Golang. Enable TCP_NODELAY => "disable Nagle's algorithm")

[1] https://github.com/golang/go/issues/57530

Re: It's always TCP_NODELAY

#198

Post by anilakar »

Sohcahtoa82 wrote: post #89
obelos wrote: post #7

Not every time. Sometimes it's DNS.

I chuckle whenever I see this meme, because in my experience, the issue is usually DHCP.

But it's usually DHCP that sets the wrong DNS servers.

It's funny that some folks claim DNS outage is a legitimate issue in systems whose both ends they control. I get it; reimplementing functionality is rarely a good sign, but since you already know your own addresses in the first place, you should also have an internal mechanism for sharing them.

Re: It's always TCP_NODELAY

#199

Post by withinboredom »

cesarb wrote: post #170
withinboredom wrote: post #111

Earlier quoted context omitted.

I think what they meant (judging by the example you ignored) is that the table changes (even if append-only) and you don't know which version you actually have when you statically compile your own version. Thus, your syscalls might be using a newer version of the table but it a) not actually be implemented, or b) implemented with something bespoke.

> Thus, your syscalls might be using a newer version of the table but it a) not actually be implemented, That's the same case as when a syscall is later removed: it returns -ENOSYS. The correct way is to do the call normally as if it were implemented, and if it returns -ENOSYS, you know that this syscall does not exist in the currently running kernel, and you should try something else. That is the same no matter whet…

Every kernel I’ve ever used has been different from an upstream kernel, with custom patches applied. It’s literally open source, anyone can do anything to it that they want. If you are using libc, you’d have a reasonable expectation not to need to know the details of those changes. If you call the kernel directly via syscall, then yeah, there is nothing you can do about someone making modifications to open source software.

Re: It's always TCP_NODELAY

#200

Post by Culonavirus »

somat wrote: post #73

What about the opposite, disable delayed acks. The problem is the pathological behavior when tinygram prevention interacts with delayed acks. There is an exposed option to turn off tinygram prevention(TCP_NODELAY), how would you tun off delayed acks instead? Say if you wanted to benchmark all four combinations and see what works best. doing a little research I found: linux has the TCP_QUICKACK socket option but you h…

Apparently you have time to "do a little research" but not to read the entire article you're reacting to? It specifically mentions TCP_QUICKACK.
Post reply on HN