Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's always TCP_NODELAY

brooker.co.za

Post reply on HN
Print view View original

131–140 of 276 posts

Re: It's always TCP_NODELAY

#131

Post by mjb »

djha-skin wrote: post #121
niutech wrote: post #102

Earlier quoted context omitted.

Shouldn't QUIC ( https://en.wikipedia.org/wiki/QUIC ) solve the TCP issues like latency?

Quic is mostly used between client and data center, but not between two datacenter computers. TCP is the better choice once inside the datacenter. Reasons: Security Updates Phones run old kernels and new apps. So it makes a lot of sense to put something that needs updated a lot like the network stack into user space, and quic does well here. Data center computers run older apps on newer kernels, so it makes sense to…

Many performance-sensitive in-datacenter applications have moved away from TCP to reliable datagram protocols. Here's what that looks like at AWS: https://ieeexplore.ieee.org/document/9167399

Re: It's always TCP_NODELAY

#132

Post by ryjo »

I just ran into this this week implementing a socket library in CLIPS. I used Berkley sockets, and before that I had only worked with higher-level languages/frameworks that abstracts a lot of these concerns away. I was quite confused when Firefox would show a "connection reset by peer." It didn't occur to me it could be an issue "lower" in the stack. `tcpdump` helped me to observe the port and I saw that the server never sent anything before my application closed the connection.

Re: It's always TCP_NODELAY

#133

Post by klabb3 »

niutech wrote: post #102
pclmulqdq wrote: post #32

In a world where bandwidth was limited, and the packet size minimum was 64 bytes plus an inter-frame gap (it still is for most Ethernet networks), sending a TCP packet for literally every byte wasted a huge amount of bandwidth. The same goes for sending empty acks. On the other hand, my general position is: it's not TCP_NODELAY, it's TCP.

Shouldn't QUIC ( https://en.wikipedia.org/wiki/QUIC ) solve the TCP issues like latency?

As someone who needed high throughput and looked to QUIC because of control of buffers, I recommend against it at this time. It’s got tons of performance problems depending on impl and the API is different.

I don’t think QUIC is bad, or even overengineered, really. It delivers useful features, in theory, that are quite well designed for the modern web centric world. Instead I got a much larger appreciation for TCP, and how well it works everywhere: on commodity hardware, middleboxes, autotuning, NIC offloading etc etc. Never underestimate battletested tech.

In that sense, the lack of TCP_NODELAY is an exception to the rule that TCP performs well out of the box (golang is already doing this by default). As such, I think it’s time to change the default. Not using buffers correctly is a programming error, imo, and can be patched.

Re: It's always TCP_NODELAY

#134

Post by nh2 »

ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

Same here. I have a hobby that on any RPC framework I encounter, I file a Github issue "did you think of TCP_NODELAY or can this framework do only 20 calls per second?".

So far, it's found a bug every single time.

Some examples: https://cloud-haskell.atlassian.net/browse/DP-108 or https://github.com/agentm/curryer/issues/3

I disagree on the "not a good / bad option" though.

It's a kernel-side heuristic for "magically fixing" badly behaved applications.

As the article states, no sensible application does 1-byte network write() syscalls. Software that does that should be fixed.

It makes sense only in the case when you are the kernel sysadmin and somehow cannot fix the software that runs on the machine, maybe for team-political reasons. I claim that's pretty rare.

For all other cases, it makes sane software extra complicated: You need to explicitly opt-out of odd magic that makes poorly-written software have slightly more throughput, and that makes correctly-written software have huge, surprising latency.

John Nagle says here and in linked threads that Delayed Acks are even worse. I agree. But the Send/Send/Receive receive pattern that Nagle's Algorithm degrades is a totally valid and common use case, including anything that does pipelined RPC over TCP.

Both Delayed Acks and Nagle's Algorithm should be opt-in, in my opinion. It should be called TCP_DELAY, which you can opt-into if you can't be asked to implement basic userspace buffering.

People shouldn't /need/ to know about these. Make the default case be the unsurprising one.

Re: It's always TCP_NODELAY

#136

Post by intelVISA »

LegionMammal978 wrote: post #124
leni536 wrote: post #96

Earlier quoted context omitted.

It should be possible to use vDSO without libc, although probably a lot of work.

It's not that much work; after all, every libc needs to have its own implementation. The kernel maps the vDSO into memory for you, and gives you the base address as an entry in the auxiliary vector. But using it does require some basic knowledge of the ELF format on the current platform, in order to parse the symbol table. (Alongside knowledge of which functions are available in the first place.)

It's hard work to NOT have the damn vDSO invade your address space. Only kludge part of Linux, well, apart from Nagle's, dlopen, and that weird zero copy kernel patch that mmap'd -each- socket recv(!) for a while.

Re: It's always TCP_NODELAY

#137

Post by bobmcnamara »

nsguy wrote: post #95
ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

The logic is really for things like Telnet sessions. IIRC that was the whole motivation.

And for block writes!

The Nagler turns a series of 4KB pages over TCP into a stream of MTU sized packets, rather than a short packet aligned to the end of each page.

Re: It's always TCP_NODELAY

#138

Post by bobmcnamara »

hinkley wrote: post #66
ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

What you really want is for the delay to be n microseconds, but there’s no good way to do that except putting your own user space buffering in front of the system calls (user space works better, unless you have something like io_uring amortizing system call times)

I'd rather have portable TCP_CORK

Re: It's always TCP_NODELAY

#139

Post by carterschonwald »

nh2 wrote: post #134
ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

Same here. I have a hobby that on any RPC framework I encounter, I file a Github issue "did you think of TCP_NODELAY or can this framework do only 20 calls per second?". So far, it's found a bug every single time. Some examples: https://cloud-haskell.atlassian.net/browse/DP-108 or https://github.com/agentm/curryer/issues/3 I disagree on the "not a good / bad option" though. It's a kernel-side heuristic for "magically…

Oh hey! It’s been a while how’re you?!

Re: It's always TCP_NODELAY

#140

Post by dataflow »

01HNNWZ0MV43FF wrote: post #115
metadaemon wrote: post #37

Earlier quoted context omitted.

I'd just love a protocol that has a built in mechanism for realizing the other side of the pipe disconnected for any reason.

To re-word everyone else's comments - "Disconnected" is not well-defined in any network.

> To re-word everyone else's comments - "Disconnected" is not well-defined in any network.

Parent said disconnected pipe, not network. It's sufficiently well-definable there.

Post reply on HN