Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's always TCP_NODELAY

brooker.co.za

Post reply on HN
Print view View original

161–170 of 276 posts

Re: It's always TCP_NODELAY

#161

Post by klabb3 »

nh2 wrote: post #134
ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

Same here. I have a hobby that on any RPC framework I encounter, I file a Github issue "did you think of TCP_NODELAY or can this framework do only 20 calls per second?". So far, it's found a bug every single time. Some examples: https://cloud-haskell.atlassian.net/browse/DP-108 or https://github.com/agentm/curryer/issues/3 I disagree on the "not a good / bad option" though. It's a kernel-side heuristic for "magically…

> As the article states, no sensible application does 1-byte network write() syscalls. Software that does that should be fixed.

Yes! And worse, those that do are not gonna be “fixed” by delays either. In this day and age with fast internets, a syscall per byte will bottleneck the CPU way before it’ll saturate the network path. The cpu limit when I’ve been tuning buffers have been somewhere in the 4k-32k range for 10Gbps ish.

> Both Delayed Acks and Nagle's Algorithm should be opt-in, in my opinion.

Agreed, it causes more problems than it solves and is very outdated. Now, the challenge is rolling out such a change as smoothly as possible, which requires coordination and a lot of trivia knowledge of legacy systems. Migrations are never trivial.

Re: It's always TCP_NODELAY

#162

Post by nailer »

nailer wrote: post #107
ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

You’re right re: making delay explicit, but also crappy use the space networking tools don’t show whether no_delay is enabled on sockets. Last time I had to do some Linux stuff, maybe 10 years ago you had to write a systemtap program. I guess it’s EBNF now. But I bet the userspace tools still suck.

> use the space

Userspace. Sorry, was using voice dictation.

Re: It's always TCP_NODELAY

#163

Post by jeroenhd »

temac wrote: post #8
theamk wrote: post #2

I don't by the reasoning for never needing Nagle anymore. Sure, telnet isn't a thing today, but I bet there are still plenty of apps which do equivalent of: write(fd, "Host: ") write(fd, hostname) write(fd, "\r\n") write(fd, "Content-type: ") etc... this may not be 40x overhead, but it'd still 5x or so.

Fix the apps. Nobody expect magical perf if you do that when writing to files, even though the OS also has its own buffers. There is no reason to expect otherwise when writing to a socket and actually nagle already doesn't save you from syscall overhead.

Everybody expects magical perf if you do that when writing files. We have RAM buffers and write caches for a reason, even on fast SSDs. We expect it so much that macOS doesn't flush to disk even when you call fsync() (files get flushed to the disk's write buffer instead).

There's some overhead to calling write() in a loop, but it's certainly not as bad as when a call to write() would actually make the data traverse whatever output stream you call it on.

Re: It's always TCP_NODELAY

#165

Post by _carbyau_ »

bjourne wrote: post #122
temac wrote: post #8

Earlier quoted context omitted.

Fix the apps. Nobody expect magical perf if you do that when writing to files, even though the OS also has its own buffers. There is no reason to expect otherwise when writing to a socket and actually nagle already doesn't save you from syscall overhead.

> Fix the apps. Nobody expect magical perf if you do that when writing to files, We write to files line-by-line or even character-by-character and expect the library or OS to "magically" buffer it into fast file writes. Same with memory. We expect multiple small mallocs to be smartly coalesced by the platform.

True to a degree. But that is a singular platform wholly controlled by the OS.

Once you put packets out into the world you're in a shared space.

I assume every conceivable variation of argument has been made both for and against Nagles at this point but it essentially revolves around a shared networking resource and what policy is in place for fair use.

Nagles fixes a particular case but interferes overall. If you fix the "particular case app" the issue goes away.

Re: It's always TCP_NODELAY

#166

Post by mjevans »

hinkley wrote: post #66
ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

What you really want is for the delay to be n microseconds, but there’s no good way to do that except putting your own user space buffering in front of the system calls (user space works better, unless you have something like io_uring amortizing system call times)

It'd probably be amazing how many poorly coded games would work better if something like...

TCP_60FPSBUFFER

Would wait for ~16mS after the first packet is queued and batch the data stream up.

Re: It's always TCP_NODELAY

#167

Post by ww520 »

ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

Same here. My first job out of college was at a database company. Queries at the client side of the client-server based database were slow. It was thought the database server was slow as hardware back then was pretty pathetic. I traced it down to the network driver and found out the default setting of TCP_NODELAY was off. I looked like a hero when turning on that option and the db benchmarks jumped up.

Re: It's always TCP_NODELAY

#168

Post by caf »

gizmo686 wrote: post #157
maple3142 wrote: post #151

Not sure if this is a bit off topic or not, but I recently encountered a problem where my program are continuously calling write to a socket in a loop that loops N times, with each of them sending about a few hundred bytes of data representing an application-level message. The loop can be understanded as some "batched messages" to server. After that, the program will try to receive data from server and do some proces…

My guess would be that the server assumes that every call to recv() terminates on a message boundary. With TCP_NODELAY and small messages, this works out fine. Every message is contained in a single packet, and the userspace buffer being read into is large enough to contain it. As such, whenever the kernel has any data to give to userspace, it has an integer number of messages to give. Nothing requires the kernel to…

Otherwise known as the "TCP is a stream-based abstraction, not a packet-based abstraction" bug.

A related one is failing to process the second of two complete commands that happen to arrive in the same recv() call.

Re: It's always TCP_NODELAY

#169

Post by ramblemonkey »

What if we changed the kernel or tcp stack to hold on to the packet for only a short time before sending it out. This could allow you to balance the latency against the network cost of many small packets. The tcp stack could even do it dynamically if needed.

Re: It's always TCP_NODELAY

#170

Post by cesarb »

withinboredom wrote: post #111
cesarb wrote: post #84

Earlier quoted context omitted.

> It's true that Linux has a stable-ish syscall table. It's not "stable-ish", it's fully stable. Once a syscall is added to the syscall table on a released version of the official Linux kernel, it might later be replaced by a "not implemented" stub (which always returns -ENOSYS), but it will never be reused for anything else. There's even reserved space on some architectures for the STREAMS syscalls, which were AFAIK…

I think what they meant (judging by the example you ignored) is that the table changes (even if append-only) and you don't know which version you actually have when you statically compile your own version. Thus, your syscalls might be using a newer version of the table but it a) not actually be implemented, or b) implemented with something bespoke.

> Thus, your syscalls might be using a newer version of the table but it a) not actually be implemented,

That's the same case as when a syscall is later removed: it returns -ENOSYS. The correct way is to do the call normally as if it were implemented, and if it returns -ENOSYS, you know that this syscall does not exist in the currently running kernel, and you should try something else. That is the same no matter whether it's compiled statically or dynamically; even a dynamic glibc has fallback paths for some missing syscalls (glibc has a minimum required kernel version, so it does not need to have fallback paths for features introduced a long time ago).

> or b) implemented with something bespoke.

There's nothing you can do to protect against a modified kernel which does something different from the upstream Linux kernel. Even going through libc doesn't help, since whoever modified the Linux kernel to do something unexpected could also have modified the C library to do something unexpected, or libc could trip over the unexpected kernel changes.

One example of this happening is with seccomp filters. They can be used to make a syscall fail with an unexpected error code, and this can confuse the C library. More specifically, a seccomp filter which forces the clone3 syscall to always return -EPERM breaks newer libc versions which try the clone3 syscall first, and then fallback to the older clone syscall if clone3 returned -ENOSYS (which indicates an older kernel that does not have the clone3 syscall); this breaks for instance running newer Linux distributions within older Docker versions.

Post reply on HN