We need a replacement for TCP in the datacenter [pdf] • Hacker News Board

Re: We need a replacement for TCP in the datacenter [pdf]

#41

Post by teraflop » Mon, Oct 31, 2022, 4:01 AM UTC

colechristensen wrote: post #34

defrost wrote: post #5

Jumping to the end: > TCP is the wrong protocol for datacenter computing. > Every aspect of TCP’s design is wrong: there is no part worth keeping. I cannot disagree and Ousterhout argues well. > Homa offers an alternative that appears to solve all of TCP’s problems. I'm well behind the curve on protocols and now I have something to learn more about. > The best way to bring Homa into widespread usage is integrate it w…

> Every aspect of TCP’s design is wrong The driver of most of a global network of computers which has been wildly successful beyond dreams before it was real… probably deserves a better deal than “every aspect is wrong”. It has worked fanatically well and chasing the long tail of performance improvements isn’t equivalent to determining what has gotten us here is wrong.

You're cherry-picking an interpretation of a single sentence, when it should be read in the context of the preceding one: Ousterhout says every aspect of TCP's design is wrong for (modern) datacenter computing. He's not saying bad decisions were made at the time it was designed, nor even that it's badly designed for other use cases today.

The first few paragraphs of the article give even more context:

> The TCP transport protocol has proven to be phenomenally successful and adaptable. [...] It is an extraordinary engineering achievement to have designed a mechanism that could survive such radical changes in underlying technology.

> However, datacenter computing creates unprecedented challenges for TCP. [...] The datacenter environment, with millions of cores in close proximity and individual applications harnessing thousands of machines that interact on microsecond timescales, could not have been envisioned by the designers of TCP, and TCP does not perform well in this environment

Re: We need a replacement for TCP in the datacenter [pdf]

#42

Post by russellbeattie » Mon, Oct 31, 2022, 4:03 AM UTC

PixelOfDeath wrote: post #23

vore wrote: post #8

Earlier quoted context omitted.

I think re UDP the point regarding it being unreliable is that you have to design your applications to take the unreliability into account, because it does happen even if it may be infrequent: assuming that it is reliable when you can get unreliable behavior will result in correctness issues.

I see UDP more like a low level interface allowing you to build your own on top. Where you decide what packages need to be revived 100% and which one can be dropped. Basically the foundation of your very own TCP with hookers and blackjack.

The new digital television broadcasting system in the US (ATSC 3.0) is exactly this. It's all UDP, but wrapped in another layer which allows multiple virtual streams, and that's all encoded in a CDMA wireless protocol. It's bundled up at the broadcast center, sent out via the big towers, and then unwrapped and decoded on the receiver. The end result is that once the receiver chipset has stripped off the wrapper, the OS of whatever client device is consuming the broadcast just gets regular looking UDP packets filled with MPEG-TS or DASH media streams, plus web pages, ads, games, or whatever. A.k.a. blackjack and hookers. Think of it as a giant one-way WiFi network using just UDP for the packets. It's honestly pretty cool.

Re: We need a replacement for TCP in the datacenter [pdf]

#43

Post by hinkley » Mon, Oct 31, 2022, 4:05 AM UTC

colechristensen wrote: post #34

defrost wrote: post #5

Jumping to the end: > TCP is the wrong protocol for datacenter computing. > Every aspect of TCP’s design is wrong: there is no part worth keeping. I cannot disagree and Ousterhout argues well. > Homa offers an alternative that appears to solve all of TCP’s problems. I'm well behind the curve on protocols and now I have something to learn more about. > The best way to bring Homa into widespread usage is integrate it w…

> Every aspect of TCP’s design is wrong The driver of most of a global network of computers which has been wildly successful beyond dreams before it was real… probably deserves a better deal than “every aspect is wrong”. It has worked fanatically well and chasing the long tail of performance improvements isn’t equivalent to determining what has gotten us here is wrong.

My Distributed Computing professor said, “now we are going to discuss why Ethernet is a terrible protocol but we use it anyway.”

Like democracy, everything else we’ve tried is even worse.

Re: We need a replacement for TCP in the datacenter [pdf]

#44

Post by Dylan16807 » Mon, Oct 31, 2022, 4:05 AM UTC

29athrowaway wrote: post #39

DonHopkins wrote: post #22

I'd tell you a UDP joke, but you probably wouldn't get it. So here's a TCP joke: Hello, would you like to hear a TCP joke? Yes, I'd like to hear a TCP joke. OK, I'll tell you a TCP joke. OK, I'll hear a TCP joke. Are you ready to hear a TCP joke? Yes, I am ready to hear a TCP joke. OK, I'm about to send the TCP joke. It will last 10 seconds, it has two characters, it does not have a setting, it ends with a punchline.…

The handshake sequence is exaggerated. It's just usually 3 messages. The 3 initial messages establish a connection. A: I would like to tell you something. (SYN) B: I acknowledge you want to tell me something. (SYN-ACK) A: I received your acknowledgement. (ACK) After the handshake sequence is done, data transfer begins. Only then, it is possible to know that the "something" was a joke. If B answers with RST instead of…

Not even "after", you'd generally put the message in the same packet as the second ACK.

Re: We need a replacement for TCP in the datacenter [pdf]

#45

Post by fragmede » Mon, Oct 31, 2022, 4:06 AM UTC

throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

What is QUIC in your book?

Given, say, $50 million of dev time, what would you go about fixing? And in what way?

Re: We need a replacement for TCP in the datacenter [pdf]

#46

Post by mlerner » Mon, Oct 31, 2022, 4:10 AM UTC

I wrote a summary of one of the approaches for replacing TCP mentioned in the paper (called Homa) here: https://www.micahlerner.com/2021/08/15/a-linux-kernel-implem...

Re: We need a replacement for TCP in the datacenter [pdf]

#47

Post by fragmede » Mon, Oct 31, 2022, 4:10 AM UTC

I mean, part of the envoy/grpc thing is to paper over some of the shortcomings of TCP so that yes, it's still TCP underneath, but you're not setting up connections the same way. Furthermore, any improvements in the space, the envoy sidecar is well positioned to do that upgrade.

Re: We need a replacement for TCP in the datacenter [pdf]

#48

Post by adobrawy » Mon, Oct 31, 2022, 4:11 AM UTC

BirAdam wrote: post #10

Well, the key would be to develop and deploy Homa in a DC and test in implementation at scale. If it actually ameliorates the perceived shortcomings of TCP that make nothing in TCP worth keeping as this author says, then cool. My only complain with issues like this is the cost of implementation. Someone has to pay to build a DC around it or increase the cost of maintenance for several years to a decade while supporti…

Homa protocl can be deployed on the basis of existing switches (not all of them, but some of them will play well). See also https://github.com/PlatformLab/HomaModule .

In addition, a closed environment and the SDN's popularity in large data centers is a significant cost-reducing factors compared to typical IPv6 deployment.

Re: We need a replacement for TCP in the datacenter [pdf]

#49

Post by hinkley » Mon, Oct 31, 2022, 4:12 AM UTC

tptacek wrote: post #15

"We hypothesize that flow-consistent routing is responsible for virtually all of the congestion that occurs in the core of datacenter networks". Flow-consistent routing is the constraint that packets for a given TCP 4-tuple get routed through the same network path, rather than balanced across all viable paths; locking a flow to a particular path makes it unlikely that segments will be received out of order on the des…

Or, by sending the traffic over all routes, there is no way to keep one server from monopolizing all traffic, because each route is oblivious to the stress currently being experienced by all its peers. It has to set a policy using local data, not global data.

The usual failure mode for clever people thinking about software is taking their third person omniscient view of the system status and thinking they can write software that replicated what a human would do in that situation. We are still so very far from human level intuition and reasoning.

Re: We need a replacement for TCP in the datacenter [pdf]

#50

Post by KerrAvon » Mon, Oct 31, 2022, 4:14 AM UTC

tptacek wrote: post #20

mlhpdx wrote: post #16

> … it should be possible to bring it into widespread usage by integrating it with RPC frameworks. That’s a powerful word, “should”. Many software in the datacenter are almost as old as TCP itself, in whole or part. Difficult but working, they will continue to linger unless something more than six letters of aspiration is applied to reimagining and rebuilding that considerable bulk.

It's theoretically much easier to introduce a new transport inside of a DC, since you're inside the network perimeter and you'll generally have control over policy-based filtering decisions.

The context here is TFA advocating for use of higher-level message-oriented frameworks instead of raw socket APIs so that they can use a non-TCP transport without changing the application code.