Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's always TCP_NODELAY

brooker.co.za

Post reply on HN
Print view View original

61–70 of 276 posts

Re: It's always TCP_NODELAY

#61

Post by noselasd »

metadaemon wrote: post #37
pclmulqdq wrote: post #32

In a world where bandwidth was limited, and the packet size minimum was 64 bytes plus an inter-frame gap (it still is for most Ethernet networks), sending a TCP packet for literally every byte wasted a huge amount of bandwidth. The same goes for sending empty acks. On the other hand, my general position is: it's not TCP_NODELAY, it's TCP.

I'd just love a protocol that has a built in mechanism for realizing the other side of the pipe disconnected for any reason.

SCTP has hearbeats to detect that.

Re: It's always TCP_NODELAY

#62

Post by deathanatos »

mirekrusin wrote: post #23

Can't it have "if payload is 1 byte (or less than X) then wait, otherwise don't" condition?

How is what you're describing not just Nagle's algorithm?

If you mean TCP_NODELAY, you should use it with TCP_CORK, which prevents partial frames. TCP_CORK the socket, do your writes to the kernel via send, and then once you have an application level "message" ready to send out — i.e., once you're at the point where you're going to go to sleep and wait for the other end to respond, unset TCP_CORK & then go back to your event loop & sleep. The "uncork" at the end + nodelay sends the final partial frame, if there is one.

Re: It's always TCP_NODELAY

#63

Post by sophacles »

metadaemon wrote: post #37
pclmulqdq wrote: post #32

In a world where bandwidth was limited, and the packet size minimum was 64 bytes plus an inter-frame gap (it still is for most Ethernet networks), sending a TCP packet for literally every byte wasted a huge amount of bandwidth. The same goes for sending empty acks. On the other hand, my general position is: it's not TCP_NODELAY, it's TCP.

I'd just love a protocol that has a built in mechanism for realizing the other side of the pipe disconnected for any reason.

That's really really hard. For a full, guaranteed way to do this we'd need circuit switching (or circuit switching emulation). It's pretty expensive to do in packet networks - each flow would need to be tracked by each middle box, so a lot more RAM at every hop, and probably a lot more processing power. If we go with circuit establishment, its also kind of expensive and breaks the whole "distributed, decentralized, self-healing network" property of the Internet.

It's possible to do better than TCP these days, bandwidth is much much less constrained than it was when TCP was designed, but it's still a hard problem to do detection of pipe disconnected for any reason other than timeouts (which we already have).

Re: It's always TCP_NODELAY

#64

Post by sophacles »

the8472 wrote: post #42
metadaemon wrote: post #37

Earlier quoted context omitted.

I'd just love a protocol that has a built in mechanism for realizing the other side of the pipe disconnected for any reason.

If a socket is closed properly there'll be a FIN and the other side can learn about it by polling the socket. If the network connection is lost due to external circumstances (say your modem crashes) then how would that information propagate from the point of failure to the remote end on an idle connection ? Either you actively probe (keepalives) and risk false positives or you wait until you hear again from the other…

It gets even worse - routing changes causing traffic to blackhole would still be undetectable without a timeout mechanism, since probes and responses would be lost.

Re: It's always TCP_NODELAY

#65

Post by jcgrillo »

chuckadams wrote: post #46
jcgrillo wrote: post #34

Earlier quoted context omitted.

For some reason this reminded me of the "500mi email" bug [1], maybe a similar level of initial apparent absurdity? [1] https://www.ibiblio.org/harris/500milemail.html

The most absurd thing to me about the 500 mile email situation is that sendmail just happily started up and soldiered on after being given a completely alien config file. Could be read as another example of "be liberal in what you accept" going awry, but sendmail's wretched config format is really a volume of war stories all its own...

Configuration changes are one of those areas where having some kind of "are you sure? (y/n)" check can really pay off. It wouldn't have helped in this case, because there wasn't really any change management process to speak of, but we haven't fully learned the lesson yet.

Re: It's always TCP_NODELAY

#66

Post by hinkley »

ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

What you really want is for the delay to be n microseconds, but there’s no good way to do that except putting your own user space buffering in front of the system calls (user space works better, unless you have something like io_uring amortizing system call times)

Re: It's always TCP_NODELAY

#67

Post by assassinator42 »

zbowling wrote: post #52
cesarb wrote: post #28

Earlier quoted context omitted.

> Would some kind of LD_PRELOAD interception for socket(2) work? That would only work if the call goes through libc, and it's not statically linked. However, it's becoming more and more common to do system calls directly, bypassing libc; the Go language is infamous for doing that, but there's also things like the rustix crate for Rust ( https://crates.io/crates/rustix ), which does direct system calls by default.

And go is wrong for doing that, at least on Linux. It bypasses optimizations in the vDSO in some cases. On Fuchsia, we made direct syscalls not through the vDSO illegal and it was funny the hacks to go that required. The system ABI of Linux really isn't the syscall interface, its the system libc. That's because the C ABI (and the behaviors of the triple it was compiled for) and its isms for that platform are the ling…

The proliferation of Docker containers seems to go against that. Those really only work well since the kernel has a stable syscall ABI. So much so that you see Microsoft switching to a stable syscall ABI with Windows 11.

Re: It's always TCP_NODELAY

#68

Post by sophacles »

zbowling wrote: post #52
cesarb wrote: post #28

Earlier quoted context omitted.

> Would some kind of LD_PRELOAD interception for socket(2) work? That would only work if the call goes through libc, and it's not statically linked. However, it's becoming more and more common to do system calls directly, bypassing libc; the Go language is infamous for doing that, but there's also things like the rustix crate for Rust ( https://crates.io/crates/rustix ), which does direct system calls by default.

And go is wrong for doing that, at least on Linux. It bypasses optimizations in the vDSO in some cases. On Fuchsia, we made direct syscalls not through the vDSO illegal and it was funny the hacks to go that required. The system ABI of Linux really isn't the syscall interface, its the system libc. That's because the C ABI (and the behaviors of the triple it was compiled for) and its isms for that platform are the ling…

Linux is also weird because there are syscalls not supported in most (any?) libc - things like io_uring, and netlink fall into this.

Re: It's always TCP_NODELAY

#69

Post by stonemetal12 »

>To make a clearer case, let’s turn back to the justification behind Nagle’s algorithm: amortizing the cost of headers and avoiding that 40x overhead on single-byte packets. But does anybody send single byte packets anymore?

That is a bit of a strawman there. While he uses single byte packets as the worst case example, the issue as stated is any not full packet.

Re: It's always TCP_NODELAY

#70

Post by sophacles »

theamk wrote: post #2

I don't by the reasoning for never needing Nagle anymore. Sure, telnet isn't a thing today, but I bet there are still plenty of apps which do equivalent of: write(fd, "Host: ") write(fd, hostname) write(fd, "\r\n") write(fd, "Content-type: ") etc... this may not be 40x overhead, but it'd still 5x or so.

TCP_CORK handles this better than nagle tho.
Post reply on HN