Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a replacement for TCP in the datacenter [pdf]

web.stanford.edu

Post reply on HN
Print view View original

51–60 of 329 posts

Re: We need a replacement for TCP in the datacenter [pdf]

#51

Post by bigDinosaur »

throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

Your ideas are interesting, can you link to or explain a concrete example though? The idea of everything magically debugging itself doesn't apply to a single piece of software I've ever seen, so I'm curious what kind of design would lead to that being possible.

Re: We need a replacement for TCP in the datacenter [pdf]

#52

Post by ajross »

Dylan16807 wrote: post #44
29athrowaway wrote: post #39

Earlier quoted context omitted.

The handshake sequence is exaggerated. It's just usually 3 messages. The 3 initial messages establish a connection. A: I would like to tell you something. (SYN) B: I acknowledge you want to tell me something. (SYN-ACK) A: I received your acknowledgement. (ACK) After the handshake sequence is done, data transfer begins. Only then, it is possible to know that the "something" was a joke. If B answers with RST instead of…

Not even "after", you'd generally put the message in the same packet as the second ACK.

Only with specialized client software. The connect() system call doesn't return to the caller until the three-way handshake is complete, so you need a fourth packet to send useful data to the server socket.

In the other direction, where you have datacenter hardware and custom kernels, it's common to see the stack cheat and start blasting packets back to the client as soon as it gets the initial SYN, just expecting that the ACK will arrive normally.

Re: We need a replacement for TCP in the datacenter [pdf]

#53

Post by kevin_nisbet »

intelVISA wrote: post #32
wmf wrote: post #28

Earlier quoted context omitted.

You can use SCTP in a datacenter. Ousterhout el al. are surely aware of SCTP so I assume Homa is better in some way.

Forgive my ignorance but why isn't SCTP more frequently used in DCs? I know it misbehaves with home routers etc. but shouldn't be a factor here.

I suspect there are a couple of contributors.

TCP is prevalent on the internet, so you need a fairly strong motivation and benefits to adopt a second protocol. A lot of engineering also doesn't get underlying networking, so one of the successes of TCP is it's a file descriptor that you either write to or read from and magic makes it come out the other side. I've seen tech leadership on networking centric products know nothing more than you read and write and magic makes the data appear on the other side. Even on implementations that use SCTP, I've seen products that only using a single stream and mark every message as requiring in order delivery. So it was effectively what TCP offers using the SCTP protocol.

At the time TCP was also far higher performance than SCTP. This wasn't so much a protocol thing, but because TCP was getting more engineering attention, it got a lot more scheduler optimization, kernel optimizations, and hardware offload support. So in many ways I think TCP scaled better due to these optimizations, which work both on the internet and internally. And then for multi-path, most data centers didn't get truly isolated networks. So if I'm running a mixture of TCP and SCTP, I still need L2 failover everywhere, which means my multi-homed SCTP connection isn't actually path diverse. And then where beneficial over the internet, there are a few success cases of using multipath TCP extensions.

SCTP is still used quite a bit in the telco networks, but due to the above, it was quite a waste of time.

Re: We need a replacement for TCP in the datacenter [pdf]

#54

Post by Dylan16807 »

ajross wrote: post #52
Dylan16807 wrote: post #44

Earlier quoted context omitted.

Not even "after", you'd generally put the message in the same packet as the second ACK.

Only with specialized client software. The connect() system call doesn't return to the caller until the three-way handshake is complete, so you need a fourth packet to send useful data to the server socket. In the other direction, where you have datacenter hardware and custom kernels, it's common to see the stack cheat and start blasting packets back to the client as soon as it gets the initial SYN, just expecting th…

Well, even if you wait for the kernel that's only microseconds before you can go from ACK to data. No round trips necessary.

Re: We need a replacement for TCP in the datacenter [pdf]

#55

Post by Areading314 »

Dylan16807 wrote: post #38
DonHopkins wrote: post #22

I'd tell you a UDP joke, but you probably wouldn't get it. So here's a TCP joke: Hello, would you like to hear a TCP joke? Yes, I'd like to hear a TCP joke. OK, I'll tell you a TCP joke. OK, I'll hear a TCP joke. Are you ready to hear a TCP joke? Yes, I am ready to hear a TCP joke. OK, I'm about to send the TCP joke. It will last 10 seconds, it has two characters, it does not have a setting, it ends with a punchline.…

A classic, but it's really far from how TCP actually works.

It needs something like -- has anyone told a TCP joke in this vicinity in the last 10 seconds? If so I will tell the joke at a slower rate

Re: We need a replacement for TCP in the datacenter [pdf]

#56

Post by ajross »

throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

Surely most of your ideas are already being deployed in QUIC/HTTP3. It just happens inside a UDP datagram, for compatibility. Really you're not going to see any new IP protocol layers, there's too much quirky hardware on the network that wouldn't be able to handle it. If we can't even get IPv6 to work all the way to the client, we're never seeing new values for the protocol byte.

Re: We need a replacement for TCP in the datacenter [pdf]

#57

Post by _jvqe »

I think protocol churn extends well past datacenters and homa, with the need for 21st century domestic protocols to remove all of the baggage from tcp/ip completely.

TCP/IP could be offshored where the undersea fiber meets the shore and setup perimeter dataedge servers as the only TCP/IP nodes on domestic boundaries.

For a domestic protocol, it does not need to be routable and saves even more overhead (I liked the tcp joke in the comments btw).

Encapsulation over a domestic protocol would still make subscriptions to a tcp/ip service available, but IMO, domestic boundaries would be better served with a modern domestic protocol.

Hypothetically speaking, if every TLD was established on its own unique protocol in the original design, Internet 1.0 would have matured much differently. Every TLD is a .com nowadays anyways, too ambiguous to be of any use (and the .org debacle still makes me laugh. They forgot what an organization was and just blended into a .com reject. That mission statement was just toilet paper after the tug-of-war).

As it turns out, one size does not fit all with protocols. Multiprotocol networks within domestic borders without tcp/ip wouldn't change any of the benefits of a data network.

Re: We need a replacement for TCP in the datacenter [pdf]

#58

Post by wiredone »

29athrowaway wrote: post #39
DonHopkins wrote: post #22

I'd tell you a UDP joke, but you probably wouldn't get it. So here's a TCP joke: Hello, would you like to hear a TCP joke? Yes, I'd like to hear a TCP joke. OK, I'll tell you a TCP joke. OK, I'll hear a TCP joke. Are you ready to hear a TCP joke? Yes, I am ready to hear a TCP joke. OK, I'm about to send the TCP joke. It will last 10 seconds, it has two characters, it does not have a setting, it ends with a punchline.…

The handshake sequence is exaggerated. It's just usually 3 messages. The 3 initial messages establish a connection. A: I would like to tell you something. (SYN) B: I acknowledge you want to tell me something. (SYN-ACK) A: I received your acknowledgement. (ACK) After the handshake sequence is done, data transfer begins. Only then, it is possible to know that the "something" was a joke. If B answers with RST instead of…

You're missing the frame size comms that then occur following this.

Re: We need a replacement for TCP in the datacenter [pdf]

#59

Post by kevin_nisbet »

ggm wrote: post #4

Out of order delivery is fine in TCP within the window. It might be inefficient but it's not impossible, reassembly could be moved to userspace if userspace TCP was used. I have no problem with alternates to TCP in the DC with a crossbar fabric and far less loss, seems sensible. I wonder how it would play with QUIC and the session like behaviours now emerging.

There was a time when out of order packets triggered congestion handling in TCP stacks which drastically reduces performance. This is where the concern comes from. I think it's a bit out dated though, I think the newer schedulers ignore out of order delivery.

I've also seen problems on some embedded stacks, but that could easily be argued that the implementation is wrong. But I've seen things like credit card terminals break due to packet reordering.

Post reply on HN