It's always TCP_NODELAY • Hacker News Board

Re: It's always TCP_NODELAY

#81

Post by Animats » Thu, May 09, 2024, 8:36 PM UTC

somat wrote: post #73

What about the opposite, disable delayed acks. The problem is the pathological behavior when tinygram prevention interacts with delayed acks. There is an exposed option to turn off tinygram prevention(TCP_NODELAY), how would you tun off delayed acks instead? Say if you wanted to benchmark all four combinations and see what works best. doing a little research I found: linux has the TCP_QUICKACK socket option but you h…

> linux has the TCP_QUICKACK socket option but you have to set it every time you receive

Right. What were they thinking? Why would you want it off only some of the time?

Re: It's always TCP_NODELAY

#82

Post by Scubabear68 » Thu, May 09, 2024, 8:39 PM UTC

Bluecobra wrote: post #72

ironman1478 wrote: post #56

I've fixed multiple latency issues due to nagle's multiple times in my career. It's the first thing I jump to. I feel like the logic behind it is sound, but it just doesn't work for some workloads. It should be something that an engineer needs to be forced to set while creating a socket, instead of letting the OS choose a default. I think that's the main issue. Not that it's a good / bad option but that there is a se…

I agree, it has been fairly well known to disable Nagle's Algorithm in HFT/low latency trading circles for quite some time now (like > 15 years). It's one of the first things I look for.

I was setting TCP_NODELAY at Bear Stearns for custom networking code circa 1994 or so.

Re: It's always TCP_NODELAY

#83

Post by matthavener » Thu, May 09, 2024, 8:42 PM UTC

zengid wrote: post #33

Relevant Oxide and Friends podcast episode https://www.youtube.com/watch?v=mqvVmYhclAg

This was a great episode and the really drove home the importance of visualization.

Re: It's always TCP_NODELAY

#84

Post by cesarb » Thu, May 09, 2024, 8:43 PM UTC

zbowling wrote: post #60

JoshTriplett wrote: post #55

Earlier quoted context omitted.

> The system ABI of Linux really isn't the syscall interface, its the system libc. You might have reasons to prefer to use libc; some software has reason to not use libc. Those preferences are in conflict, but one of them is not automatically right and the other wrong in all circumstances. Many UNIX systems did follow the premise that you must use libc and the syscall interface is unstable. Linux pointedly did not, a…

It's true that Linux has a stable-ish syscall table. What is funny is that this caused a whole series of Samsung Android phones to reboot randomly with some apps because Samsung added a syscall at the same position someone else did in upstream linux and folks staticly linking their own libc to avoid boionc libc were rebooting phones when calling certain functions because the Samsung syscall causing kernel panics when…

> It's true that Linux has a stable-ish syscall table.

It's not "stable-ish", it's fully stable. Once a syscall is added to the syscall table on a released version of the official Linux kernel, it might later be replaced by a "not implemented" stub (which always returns -ENOSYS), but it will never be reused for anything else. There's even reserved space on some architectures for the STREAMS syscalls, which were AFAIK never on any released version of the Linux kernel.

The exception is when creating a new architecture; for instance, the syscall table for 32-bit x86 and 64-bit x86 has a completely different order.

Re: It's always TCP_NODELAY

#85

Post by voxic11 » Thu, May 09, 2024, 8:43 PM UTC

mort96 wrote: post #71

koverstreet wrote: post #38

Earlier quoted context omitted.

Like TCP keepalives?

If the feature already technically exists in TCP, it's either broken or disabled by default, which is pretty much the same as not having it.

keepalives are an optional TCP feature so they are not necessarily supported by all TCP implementations and therefor default to off even when supported.

Re: It's always TCP_NODELAY

#86

Post by gpderetta » Thu, May 09, 2024, 8:46 PM UTC

sophacles wrote: post #68

zbowling wrote: post #52

Earlier quoted context omitted.

And go is wrong for doing that, at least on Linux. It bypasses optimizations in the vDSO in some cases. On Fuchsia, we made direct syscalls not through the vDSO illegal and it was funny the hacks to go that required. The system ABI of Linux really isn't the syscall interface, its the system libc. That's because the C ABI (and the behaviors of the triple it was compiled for) and its isms for that platform are the ling…

Linux is also weird because there are syscalls not supported in most (any?) libc - things like io_uring, and netlink fall into this.

Futex for a very long time was only accessible via syscall.

Re: It's always TCP_NODELAY

#87

Post by batmanthehorse » Thu, May 09, 2024, 8:51 PM UTC

somat wrote: post #73

What about the opposite, disable delayed acks. The problem is the pathological behavior when tinygram prevention interacts with delayed acks. There is an exposed option to turn off tinygram prevention(TCP_NODELAY), how would you tun off delayed acks instead? Say if you wanted to benchmark all four combinations and see what works best. doing a little research I found: linux has the TCP_QUICKACK socket option but you h…

In CentOS/RedHat you can add `quickack 1` to the end of a route to tell it to disable delayed acks for that route.

Re: It's always TCP_NODELAY

#88

Post by benreesman » Thu, May 09, 2024, 8:56 PM UTC

Nagle and no delay are like like 90+% of the latency bugs I’ve dealt with.

Two reasonable ideas that mix terribly in practice.

Re: It's always TCP_NODELAY

#89

Post by Sohcahtoa82 » Thu, May 09, 2024, 8:57 PM UTC

obelos wrote: post #7

Not every time. Sometimes it's DNS.

I chuckle whenever I see this meme, because in my experience, the issue is usually DHCP.

Re: It's always TCP_NODELAY

#90

Post by JoshTriplett » Thu, May 09, 2024, 8:59 PM UTC

tedunangst wrote: post #76

JoshTriplett wrote: post #55

Earlier quoted context omitted.

> The system ABI of Linux really isn't the syscall interface, its the system libc. You might have reasons to prefer to use libc; some software has reason to not use libc. Those preferences are in conflict, but one of them is not automatically right and the other wrong in all circumstances. Many UNIX systems did follow the premise that you must use libc and the syscall interface is unstable. Linux pointedly did not, a…

The complication with the linux syscall interface is that it turns the worse is better up to 11. Like setuid works on a per thread basis, which is seriously not what you want, so every program/runtime must do this fun little thread stop and start and thunk dance.

Yeah, agreed. One of the items on my long TODO list is adding `setuid_process` and `setgid_process` and similar, so that perhaps a decade later when new runtimes can count on the presence of those syscalls, they can stop duplicating that mechanism in userspace.