Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's Always TCP_NODELAY

brooker.co.za

Post reply on HN
Print view View original

101–110 of 186 posts

Re: It's Always TCP_NODELAY

#101

Post by rwmj »

mgaunard wrote: post #96
rwmj wrote: post #79

I'm surprised the article didn't also mention MSG_MORE. On Linux it hints to the kernel that "more is to follow" (when sending data on a socket) so it shouldn't send it just yet. Maybe you need to send a header followed by some data. You could copy them into one buffer and use a single sendmsg call, but it's easier to send the header with MSG_MORE and the data in separate calls. (io_uring is another method that helps…

you can already send fragmented data in one system call without copying it to a single buffer.

Indeed you can, but we've found it useful to use MSG_MORE when using state machines, where different states are responsible for different parts of the reply. (Plenty of examples in states*.c here: https://gitlab.com/nbdkit/libnbd/-/tree/master/generator?ref...)

Re: It's Always TCP_NODELAY

#102

Post by TZubiri »

vsgherzi wrote: post #7

https://oxide-and-friends.transistor.fm/episodes/mr-nagles-w... oxide and friends episode on it! It's quite good

Very on brand, oxide's core proposition is to actually invent a new (server) os+hardware, so they question/polish many of the traditional protocols and standards from the golden era.

Re: It's Always TCP_NODELAY

#103

Post by mort96 »

mgaunard wrote: post #97
mort96 wrote: post #90

Earlier quoted context omitted.

It's the main reason why I use websockets for a whole lot of things. I don't wanna build my own message chunking layer on top of TCP every time.

WebSocket is full of web-tech silliness; you'd be better off doing your own framing.

Well, it also has the advantage of providing pretty decent encryption for free through WSS.

But yeah, where that's unnecessary, it's probably just as easy to have a 4-byte length prefix, since TCP handles the checksum and retransmit and everything for you.

Re: It's Always TCP_NODELAY

#104

Post by mgaunard »

rwmj wrote: post #101
mgaunard wrote: post #96

Earlier quoted context omitted.

you can already send fragmented data in one system call without copying it to a single buffer.

Indeed you can, but we've found it useful to use MSG_MORE when using state machines, where different states are responsible for different parts of the reply. (Plenty of examples in states*.c here: https://gitlab.com/nbdkit/libnbd/-/tree/master/generator?ref... )

Doing more system calls isn't really a good idea for performance.

Also if you're doing asynchronous writes you typically can only have one write in-flight at any time, you should aggregate all other buffers while that happens.

Though arguably asynchronous writes are often undesired due to the complexity of doing flow-control with them.

Re: It's Always TCP_NODELAY

#105

Post by mgaunard »

mort96 wrote: post #103
mgaunard wrote: post #97

Earlier quoted context omitted.

WebSocket is full of web-tech silliness; you'd be better off doing your own framing.

Well, it also has the advantage of providing pretty decent encryption for free through WSS. But yeah, where that's unnecessary, it's probably just as easy to have a 4-byte length prefix, since TCP handles the checksum and retransmit and everything for you.

It's just a standard TLS layer, works with any TCP protocol, nothing WebSocket-specific in it.

You should ideally design your messages to fit within a single Ethernet packet, so 2 bytes is more than enough for the size. Though I have sadly seen an increasing amount of developers send arbitrarily large network messages and not care about proper design.

Re: It's Always TCP_NODELAY

#106

Post by BikiniPrince »

linohh wrote: post #89
mikestorrent wrote: post #64

Earlier quoted context omitted.

What did you still need to connect with 10mbit half duplex in 2014? I had gigabit to the desktop for a relatively small company in 2007, by 2014 10mb was pretty dead unless you had something Really Interesting connected....

Technical debt goes hard, I had a discussion with a facilities guy why they never got around to ditch the last remnants of token ring in an office park. Fortunately in 2020 they had plenty of time to rip that stuff out without disturbing facility operation. Building automation, security and so on often lives way longer than you'd dare planning.

Everyone is forgetting the no delay is per application and not a system configuration. Yep, old things will still be old and that’s ok. That new fangled packet farter will need to set no delay which is a default in many scenarios. This article reminds us it is a thing and especially true for home grown applications.

Re: It's Always TCP_NODELAY

#107

Post by Hikikomori »

saltcured wrote: post #46
gerdesj wrote: post #40

Earlier quoted context omitted.

I think you are confusing network layers and their functionality. "CSMA is no longer necessary on Ethernet today because all modern connections are point-to-point with only two "hosts" per channel." Ethernet really isn't ptp. You will have a switch at home (perhaps in your router) with more than two ports on it. At layer 1 or 2 how do you mediate your traffic, without CSMA? Take a single switch with n ports on it, wh…

In modern ethernet, there is also flow-control via the PAUSE frame. This is not for collisions at the media level, but you might think of it as preventing collisions at the buffer level. It allows the receiver to inform the sender to slow down, rather than just dropping frames when its buffers are full.

Its not really used in normal networks.

Re: It's Always TCP_NODELAY

#108

Post by withinboredom »

silisili wrote: post #68
AnthonyMouse wrote: post #54

Earlier quoted context omitted.

The programs that need it are mostly the ones nobody is maintaining. TCP_NODELAY can also make fingerprinting easier in various ways which is a reason to make it something you have to ask for.

> The programs that need it are mostly the ones nobody is maintaining Yes, as I mentioned, it should be kept around for this but off by default. Make it a sysctl param, done. > TCP_NODELAY can also make fingerprinting easier in various ways which is a reason to make it something you have to ask for Only because it's on by default for no real reason. I'm saying the default should be off.

Nagles algorithm does really well when you're on shitty wifi.

Applications also don't know the MTU (the size of packets) on the interface they're using. Hell, they probably don't even know which interface they're using! This is all abstracted away. So, if you're on a network with a 14xx MTU (such as a VPN), assuming an MTU of 1500 means you'll send one full packet and then a tiny little packet after that. For every one packet you think you're sending!

Nagle's algorithm lets you just send data; no problem. Let the kernel batch up packets. If you control the protocol, just use a design that prevents Delayed ACK from causing the latency. IE, the "OK" from Redis.

Re: It's Always TCP_NODELAY

#109

Post by ahoka »

gerdesj wrote: post #42
anonymousiam wrote: post #41

Earlier quoted context omitted.

It's P2P as far as the physical layer (L1) is concerned. Usually, full duplex requires two separate channels. The introduction of a hybrid on each end allows the use of the same channel at the same time. Some progress has been made in doing the same thing with radio links, but it's harder. Nagle's algorithm is somewhat intertwined with the backoff timer in the sense that it prevents transmitting a packet until some c…

Sorry? Ethernet has had the concept of full duplex for several decades and I have no idea what you mean by: "hybrid on each end allows the use of the same channel at the same time." The physical electrical connections between a series of ethernet network ports (switch or end point - it doesn't matter) are mediated by CSMA. No idea why you are mentioning radios. That's another medium.

CSMA is last used with 10Mbit ethernet, so that’s why radios are only relevant.

Re: It's Always TCP_NODELAY

#110

Post by tecleandor »

miduil wrote: post #6
eieio wrote: post #2

I found this article while debugging some networking delays for a game that I'm working on. It turns out that in my case it wasn't TCP_NODELAY - my backend is written in go, and go sets TCP_NODELAY by default! But I still found the article - and in particular Nagle's acknowledgement of the issues! - to be interesting. There's a discussion from two years ago here: https://news.ycombinator.com/item?id=40310896 - but I…

There is also a good write-up [0] by Julia Evans. We ran into this with DICOM storescp, which is a chatty protocol and TCP_NODELAY=1 makes the throughput significantly better. Since DICOM is often used in a LAN, that default just makes it unnecessarily worse. [0]: https://jvns.ca/blog/2015/11/21/why-you-should-understand-a-... [1]: https://news.ycombinator.com/item?id=10607422

Oh, DICOM really like to talk back and forth... I guess that nowadays it should be better with all the WEB/REST versions of the protocol.
Post reply on HN