Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's Always TCP_NODELAY

brooker.co.za

Post reply on HN
Print view View original

151–160 of 186 posts

Re: It's Always TCP_NODELAY

#151

Post by paulsutter »

jandrese wrote: post #149
paulsutter wrote: post #147

Earlier quoted context omitted.

For serving web pages, TCP is only used by legacy servers. The fundamental congestion control issue is that after you drop to half, the window is increased by /one packet/, which for all sorts of artificial reasons is about 1500 bytes. Which means the performance gets worse and worse the greater the bandwidth-delay product (which have increased by tens of orders of magnitude). Not to mention head-of-line blocking etc…

TCP Reno fixed that problem. QUIC is more about sending more parts of the page in parallel. It does do its own flow control, but that's not where it gets the majority of the improvement.

TCP Reno Vegas etc all addressed congestion control with various ideas, but were all doomed by the academic downward spiral pissing contest.

QUIC is real and works great, and they sidestepped all of that and just built it and tuned it and has basically won. As for QUIC "sending more parts of the page in parallel" yes thats what I referred to re head of line blocking in TCP.

Re: It's Always TCP_NODELAY

#152

Post by reorder9695 »

anonymousiam wrote: post #150
reorder9695 wrote: post #118

Earlier quoted context omitted.

Is avoiding a collision always preferable? CSMA/CA has significant overhead (backoff period) for every single frame sent, on a less congested line CSMA/CD has less overhead.

But CSMA/CD also has a backoff period.

CSMA/CD only requires that you back off if there actually is a collision. CSMA/CA additionally requires that for every frame sent, after sensing the medium as clear, that you wait for a random amount of time before sending it to avoid collisions. If the medium is frequently clear, CA will still have the overhead of this initial wait where CD will not.

Re: It's Always TCP_NODELAY

#153

Post by drewg123 »

paulsutter wrote: post #151
jandrese wrote: post #149

Earlier quoted context omitted.

TCP Reno fixed that problem. QUIC is more about sending more parts of the page in parallel. It does do its own flow control, but that's not where it gets the majority of the improvement.

TCP Reno Vegas etc all addressed congestion control with various ideas, but were all doomed by the academic downward spiral pissing contest. QUIC is real and works great, and they sidestepped all of that and just built it and tuned it and has basically won. As for QUIC "sending more parts of the page in parallel" yes thats what I referred to re head of line blocking in TCP.

There is nothing magic about the congestion control in QUIC. It shares a lot with TCP BBR.

Unlike TLS over TCP, QUIC is still not able to be offloaded to NICs. And most stacks are in userspace. So it is horrifically expensive in terms of watts/byte or cycles/byte sent for a CDN workload (something like 8x as a expensive the last time I looked), and its primarily used and advocated for by people who have metrics for latency, but not server side costs.

Re: It's Always TCP_NODELAY

#154

Post by medoc »

Around 1999, I was testing a still young MySQL as an INFORMIX replacement, and network queries needed a very suspect and quite exact 100 mS. A bug report and message to mysql@lists.mysql.com, and this is how MySQL got to set TCP_NODELAY on its network sockets...

Re: It's Always TCP_NODELAY

#155

Post by CamperBob2 »

sundbry wrote: post #146
Gibbon1 wrote: post #72

Earlier quoted context omitted.

Reminds me of trying to do IoT stuff in hospitals before IoT was a thing. Send exactly one 205 byte packet. How do you really know? I can see it go out on a scope. And the other end receives a packet with bytes 0-56. Then another packet with bytes 142-204. Finally a packet a 200ms later with bytes 57-141. FfffFFFFffff You!

If only there was some sort of User Datagram Protocol where you could send specifically tuned packets like this.

Those who do not understand TCP are doomed to reimplement it with UDP.

The same is true of those who do understand it.

Re: It's Always TCP_NODELAY

#156

Post by CamperBob2 »

littlecranky67 wrote: post #78
CamperBob2 wrote: post #48

Earlier quoted context omitted.

If by "latency" you mean a hundred milliseconds or so, that's one thing, but I've seen Nagle delay packets by several seconds. Which is just goofy, and should never have been enabled by default, given the lack of an explicit flush function. A smarter implementation would have been to call it TCP_MAX_DELAY_MS, and have it take an integer value with a well-documented (and reasonably low) default.

It delays one RTT, so if you have seen seconds of delays that means your TCP ACK packages were received seconds later for whatever reason (high load?). Decreasing latency in that situation would WORSEN the situation.

Maybe, maybe not, whatever.

I was testing some low-bandwidth voice chat code using two unloaded PCs sitting on the same desk. I nearly jumped out of my skin when "HELLO, HELLO?" came through a few seconds late, at high volume, after I had already concluded it wasn't working. After ruling out latency on the audio side, TCP_NODELAY solved the problem.

All respect to Animats, but whoever thought this should be the default behavior of TCP/IP had rocks in their head, and/or were solving a problem that had a better solution that they just didn't think of at the time.

Re: It's Always TCP_NODELAY

#157

Post by saltcured »

toast0 wrote: post #59
saltcured wrote: post #46

Earlier quoted context omitted.

In modern ethernet, there is also flow-control via the PAUSE frame. This is not for collisions at the media level, but you might think of it as preventing collisions at the buffer level. It allows the receiver to inform the sender to slow down, rather than just dropping frames when its buffers are full.

At least in networks I've used, it's better for buffers to overflow than to use PAUSE. Too many switches will get a PAUSE frame from port X and send it to all the ports that send packets destined for port X. Then those ports stop sending all traffic for a while. About the only useful thing is if you can see PAUSE counters from your switch, you can tell a host is unhealthy from the switch whereas inbound packet overfl…

Sadly, I'm not too surprised to hear that. I wish we had more rapid iteration to improve such capabilities for real world use cases.

Things like back pressure and flow control are very powerful systems concepts, but intrinsically need there to be an identifiable flow to control! Our systems abstractions that multiplex and obfuscate flows are going to be unable to differentiate which application flow is the one that needs back pressure, and paint too-wide brush.

In my view, the fundamental problem is we're all trying to "have our cake and eat it". We expect our network core to be unaware of the edge device and application goals. We expect to be able to saturate an imaginary channel between two edge devices without any prearrangement, as if we're the only network users. We also expect our sparse and async background traffic to somehow get through promptly. We expect fault tolerance and graceful degradation. We expect fairness.

We don't really define or agree what is saturation, what is prompt, what is graceful, or what is fair... I think we often have selfish answers to these questions, and this yields a tragedy of the commons.

At the same time, we have so many layers of abstraction where useful flow information is effectively hidden from the layers beneath. That is even before you consider adversarial situations where the application is trying to confuse the issue.

Re: It's Always TCP_NODELAY

#158

Post by dan-robertson »

A few random thoughts:

1. Perhaps on more modern hardware the thing to do with badly behaved senders is not ‘hang on to unfull packets for 40ms’ but another policy could still work, e.g. eagerly send the underfilled packet, but wait the amount of time it would take to send a full packet (and prioritize sending other flows) before sending the next underfull packet.

2. In Linux there are packets and then there are (jumbo)packets. The networking stack has some per-packet overhead so much work is done to have it operate on bigger batches and then let the hardware (or a last step in the OS) do segmentation. It’s always been pretty unclear to me how all these packet-oriented things (Nagle’s algorithm, tc, pacing) interact with jumbo packets and the various hardware offload capabilities.

3. This kind of article comes up a lot (mystery 40ms latency -> set TCP_NODELAY). In the past I’ve tried to write little test programs in a high level language to listen on tcp and respond quickly, and in some cases (depending on response size) I’ve seen strange ~40ms latencies despite TCP_NODELAY being set. I didn’t bother looking in huge detail (eg I took a strace and tcpdump but didn’t try to see non-jumbo packets) and failed to debug the cause. I’m still curious what may have caused this?

Re: It's Always TCP_NODELAY

#159

Post by mkipper »

derriz wrote: post #132
mgaunard wrote: post #127

Earlier quoted context omitted.

TCP is not message-oriented. Retransmitted bytes can be at arbitrary offsets and do not need to align with the way the original transmission was fragmented or even an earlier retransmission.

I don’t understand your point here or maybe our understanding of the admittedly vague term “message oriented” differs. I’m not suggesting exposing retransmission, fragmentation, etc to the API user. The sender provides n bytes of data (a message) to the network stack. The receiver API provides the user with the block of n bytes (the message) as part of an atomic operation. Optionally the sender can be provided with n…

I think you're misunderstanding their point.

Your API is constrained by the actual TCP protocol. Even if the sender uses this message-oriented TCP API, the receiver can't make any guarantees that a packet they receive lines up with a message boundary, contains N messages, etc etc, due to how TCP actually works in the event of dropped packets and retransmissions. The receiver literally doesn't have the information needed to do that, and it's impossible for the receiver to reconstruct the original message sequence from the sender. You could probably re-implement TCP with retransmission behaviour that gives you what you're looking for, but that's not really TCP anymore.

This is part of the motivation for protocols like QUIC. Most people agree that some hybrid of TCP and UDP with stateful connections, guaranteed delivery and discrete messages is very useful. But no matter how much you fiddle with your code, neither TCP or UDP are going to give you this, which is why we end up with new protocols that add TCP-ish behaviour on top of UDP.

Re: It's Always TCP_NODELAY

#160

Post by dan-robertson »

eieio wrote: post #61
sail0rm00n wrote: post #18

Earlier quoted context omitted.

Any details on the game you’ve been working on? I’ve been really enjoying Ebitengine and Golang for game dev so would love to read about what you’ve been up to!

I've been playing with multiplayer games that run over SSH; right now I'm trying to push the framerate on the games as high as I can, which is what got me thinking about my networking stack. I mostly use go these days for the backend for my multiplayer games, and in this case there's also some good tooling for terminal rendering and SSH stuff in go, so it's a nice choice. (my games are often pretty weird, I understan…

Two things that can have a big impact on SSH throughput are cipher choice and the hardcoded receive buffer size. These are described in the fork https://github.com/rapier1/hpn-ssh

Maybe that will be useful for thinking about workarounds or maybe you can just use hpn-ssh.

Post reply on HN