So... basically another reinvention of UDP? I'm not entirely sure what "dataceter computing" is supposed to mean, given that a datacenter can be hosting machines for a very wide array of applications with very different network requirements.
We need a replacement for TCP in the datacenter [pdf]
71–80 of 329 posts
Re: We need a replacement for TCP in the datacenter [pdf]
#72I'd tell you a UDP joke, but you probably wouldn't get it. So here's a TCP joke: Hello, would you like to hear a TCP joke? Yes, I'd like to hear a TCP joke. OK, I'll tell you a TCP joke. OK, I'll hear a TCP joke. Are you ready to hear a TCP joke? Yes, I am ready to hear a TCP joke. OK, I'm about to send the TCP joke. It will last 10 seconds, it has two characters, it does not have a setting, it ends with a punchline.…
Hello, would you like to hear a UDP joke?
No
Knock, knock!
Re: We need a replacement for TCP in the datacenter [pdf]
#73Out of order delivery is fine in TCP within the window. It might be inefficient but it's not impossible, reassembly could be moved to userspace if userspace TCP was used. I have no problem with alternates to TCP in the DC with a crossbar fabric and far less loss, seems sensible. I wonder how it would play with QUIC and the session like behaviours now emerging.
> • In-order packet delivery
This is a bit disingenuous, since it’s not the wire protocol but the kernel API that maintains the in order abstraction. With Jumbo packets you can still push a mountain of data without tripping up on “in order packet delivery”
As developers we like this in order delivery to userspace because it vastly simplifies the code. We make up for the inefficiencies by processing dozens of hundreds of streams in parallel. We aren’t going to give that up just because the wire protocol changes.
Re: We need a replacement for TCP in the datacenter [pdf]
#74It's interesting that everyone (including the author) talks about UDP as a lossy protocol, but it doesn't seem that UDP drops actually occur on a routine basis anywhere. The UDP-based DDOS attacks seem to prove that; if UDP really was being dropped, those DDOS attacks wouldn't be so problematic. That said, it's an interesting read. TCP is inefficient, but that inefficiency has been patched/masked by hardware solution…
Nothing deliberately drops UDP packets, but packets of all sorts get dropped when there's congestion.
Which is why protocols like TCP are built on back pressure - so you don’t keep making the exact same mistake in a tight loop. Happy path behavior doesn’t matter when the worst case or even median case are nonfunctional.
Re: We need a replacement for TCP in the datacenter [pdf]
#75Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…
What is QUIC in your book? Given, say, $50 million of dev time, what would you go about fixing? And in what way?
In addition to QUIC, KCP [1] is another reliable low-latency protocol that sits on top of UDP that might be interesting. And unlike RFC 9000/9001 (QUIC), encryption is optional. I haven't really seen it mentioned much outside of primarily China-focused projects, like V2Ray [2], but there is also some English information in their Git repo [2].
[1]: https://github.com/skywind3000/kcp>
[2]: https://www.v2fly.org/en_US/>
[3]: https://github.com/skywind3000/kcp/blob/master/README.en.md>
Re: We need a replacement for TCP in the datacenter [pdf]
#76Earlier quoted context omitted.
Surely most of your ideas are already being deployed in QUIC/HTTP3. It just happens inside a UDP datagram, for compatibility. Really you're not going to see any new IP protocol layers, there's too much quirky hardware on the network that wouldn't be able to handle it. If we can't even get IPv6 to work all the way to the client, we're never seeing new values for the protocol byte.
Don’t the hyperscaled cloud providers run totally segmented networks? What’s stopping them from using something proprietary internally and just exposing TCP at the end for termination of client connections?
Google already does that.
Re: We need a replacement for TCP in the datacenter [pdf]
#77Re: We need a replacement for TCP in the datacenter [pdf]
#78Jumping to the end: > TCP is the wrong protocol for datacenter computing. > Every aspect of TCP’s design is wrong: there is no part worth keeping. I cannot disagree and Ousterhout argues well. > Homa offers an alternative that appears to solve all of TCP’s problems. I'm well behind the curve on protocols and now I have something to learn more about. > The best way to bring Homa into widespread usage is integrate it w…
> Every aspect of TCP’s design is wrong The driver of most of a global network of computers which has been wildly successful beyond dreams before it was real… probably deserves a better deal than “every aspect is wrong”. It has worked fanatically well and chasing the long tail of performance improvements isn’t equivalent to determining what has gotten us here is wrong.
"Specifically, Homa aims to replace TCP, which was designed in the era before modern data center environments existed. Consequently, TCP doesn’t take into account the unique properties of data center networks (like high-speed, high-reliability, and low-latency). Furthermore, the nature of RPC traffic is different - RPC communication in a data center often involve enormous amounts of small messages and communication between many different machines."[0]
0: https://www.micahlerner.com/2021/08/15/a-linux-kernel-implem...
Re: We need a replacement for TCP in the datacenter [pdf]
#79"We hypothesize that flow-consistent routing is responsible for virtually all of the congestion that occurs in the core of datacenter networks". Flow-consistent routing is the constraint that packets for a given TCP 4-tuple get routed through the same network path, rather than balanced across all viable paths; locking a flow to a particular path makes it unlikely that segments will be received out of order on the des…
Or, by sending the traffic over all routes, there is no way to keep one server from monopolizing all traffic, because each route is oblivious to the stress currently being experienced by all its peers. It has to set a policy using local data, not global data. The usual failure mode for clever people thinking about software is taking their third person omniscient view of the system status and thinking they can write s…
Ultimately one server cannot inject more than one link worth of traffic (e.g. 100 Gbps) into the network which is a tiny fraction of total capacity. Researchers have gotten really good results with "spray and pray" for sub-RTT flows combined with latency and queue depth feedback for multi-RTT flows.
Re: We need a replacement for TCP in the datacenter [pdf]
#80"We hypothesize that flow-consistent routing is responsible for virtually all of the congestion that occurs in the core of datacenter networks". Flow-consistent routing is the constraint that packets for a given TCP 4-tuple get routed through the same network path, rather than balanced across all viable paths; locking a flow to a particular path makes it unlikely that segments will be received out of order on the des…
Or, by sending the traffic over all routes, there is no way to keep one server from monopolizing all traffic, because each route is oblivious to the stress currently being experienced by all its peers. It has to set a policy using local data, not global data. The usual failure mode for clever people thinking about software is taking their third person omniscient view of the system status and thinking they can write s…
And we could totally construct systems that take some approximation of a global internet state into local routing decisions. But that might devalue some incumbent player's position in the market (or create a new privileged set of players) so even if we made a POC, it wouldn't get adopted.