Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a replacement for TCP in the datacenter [pdf]

web.stanford.edu

Post reply on HN
Print view View original

171–180 of 329 posts

Re: We need a replacement for TCP in the datacenter [pdf]

#171

Post by kortilla »

throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo.

This is severe bullshit on two fronts:

- there is an immediate return on value - Google was driving this a decade+ ago for improvements in the data center (things like doubled+cancelable rpc, tcp cubic, quic, etc)

- academia constantly attempts to make these improvements as well because researchers are super incentivized to dethrone tcp for the glory. There are constant attempts to re-invent various layers (IP, tcp, the non existent upper layers of the OSI, etc) that come out of academic conferences every year.

The reason we’re still here is because our current stacks have been heavily optimized and tooled for production workloads. NICs can transparently re-assemble TCP segments for the OS and they can segment before transmit. You have to have a damn good value prop to throw away everything from software and hardware to careers and curriculum. It has to be a shitload better than the security nightmare of “return back each layer of the stack”.

Re: We need a replacement for TCP in the datacenter [pdf]

#172

Post by bheadmaster »

bruce343434 wrote: post #153
josephg wrote: post #125

> The data model for TCP is a stream of bytes. However, this is not the right data model for most datacenter applications. Datacenter applications typically exchange discrete messages to implement remote procedure calls This isn't just a datacenter problem. Every single network protocol I've ever created or implemented is message based, not stream based. Every messaging system. Every video game. Every RPC transport.…

What’s the issue with fitting messages in streams?

One of the issues I can think of is head-of-line blocking [0].

If you're sending messages of different priorities over the same channel, an error in sending a low-priority message, high-priority messages will have to wait until the low-priority message is properly re-transmitted.

[0] https://en.wikipedia.org/wiki/Head-of-line_blocking

https://en.wikipedia.org/wiki/Head-of-line_blocking

Re: We need a replacement for TCP in the datacenter [pdf]

#173

Post by Aissen »

josephg wrote: post #125

> The data model for TCP is a stream of bytes. However, this is not the right data model for most datacenter applications. Datacenter applications typically exchange discrete messages to implement remote procedure calls This isn't just a datacenter problem. Every single network protocol I've ever created or implemented is message based, not stream based. Every messaging system. Every video game. Every RPC transport.…

Because most protocols can handle message loss, with retransmit and proper ordering ? And we haven't started to talk about congestion yet… TCP is useful, and while I'd like to see a message get rid of one (or more) of those constraints to go with a a custom protocol, I feel like they'd be re-implementing the features in the end because these are very useful properties to have…

Edit: the proposal in the article is actually quite, sensible, but requires redesigning your apps… And I'd like to see how it performs: TCP is a hugely optimized beast (when it works well), with hardware offloads, kernel optimizations, etc.

Re: We need a replacement for TCP in the datacenter [pdf]

#174

Post by gjulianm »

throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

> We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years.

I work in a company that builds network troubleshooting/observability tools and we have some pretty experienced analysts to tell you what's wrong with the network. With that context, your idea of having any tool automatically diagnosing network issues is a pipe dream.

The problem with networks is that they're very complex systems, with multiple elements along the way, made by different manufacturers, often with different owners, failures aren't always easily reproducible, and with human configuration (and therefore errors) almost every step of the way. Even if a tool that "returns each layer of the stack" would be useful, it still would be far from enough to diagnose issues.

Re: We need a replacement for TCP in the datacenter [pdf]

#175

Post by GTP »

idlehand wrote: post #139
bsder wrote: post #109

Earlier quoted context omitted.

> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. And yet every time hardware designers get the chance they redesign Ethernet and IPv4--poorly. See: HDMI 2.0+, USB 3.0+, Thunderbolt 3.0+, etc. My suspicion is that this paper works fine beween pairs of peers and immediately goes straight to hell after that. It is extremely suspicious that there is…

Never thought about that before. Ethernet supports extremely high levels of data transmission. ISB C for intrgrated charging ND data transfer makes sense, but why are there HDMI cables?

I think it would be an overkill to use a networking protocol to connect exactly two devices. Plus if you have something specific to video stream transfer you could maybe do some optimization specific to that use case, although I can't think of any at the moment.

Re: We need a replacement for TCP in the datacenter [pdf]

#176

Post by cdogl »

throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

I’ll defer to experts on the network-layer problems but im not sure what you see as the problem with converging on HTTP. It’s awkward and inelegant, but as an a backend application developer I never feel like it gets in my way.

Re: We need a replacement for TCP in the datacenter [pdf]

#177

Post by kortilla »

Luker88 wrote: post #166
kanwisher wrote: post #145

Earlier quoted context omitted.

Why don’t you actually build gasp a prototype before asking for money

yeah, thank you for the kind comments about not needing money (aka: my time as no value) and asking me to build the prototype with irony. As I said, the project was started a few years back, and since I did not have the time to work on it, maybe it means my life does not give me the time and money to build this on the side. But I'll always find it funny how half of the people go "you need to have solid theory proofs…

> yeah, thank you for the kind comments about not needing money (aka: my time as no value) and asking me to build the prototype with irony

The world (read: the investment community) doesn’t care about your time though. If you are going to pitch an infrastructure project that should work in software without a working demo, you’re going to have to take a huge haircut on valuation.

If you want to be successful with a theory and proofs, join academia and publish them. If you want to get investors for infrastructure, make something that works.

Academia churns out “protocols on paper” every year that go absolutely nowhere. You need to differentiate yourself if you’re looking for more than a research grant.

Re: We need a replacement for TCP in the datacenter [pdf]

#178

Post by yaantc »

josephg wrote: post #125

> The data model for TCP is a stream of bytes. However, this is not the right data model for most datacenter applications. Datacenter applications typically exchange discrete messages to implement remote procedure calls This isn't just a datacenter problem. Every single network protocol I've ever created or implemented is message based, not stream based. Every messaging system. Every video game. Every RPC transport.…

SCTP [1] is there to provide a reliable message based protocol. And it does work inside a datacenter. The issue is outside the datacenter: it doesn't work reliably across the Internet due to middle boxes dumping anything not TCP or UDP...

But inside a controlled environment like a datacenter, it works. It's been used in the telecommunication world to carry control messages in the radio access and core networks for example. So it's been tested at scale for critical applications.

[1] https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...

Re: We need a replacement for TCP in the datacenter [pdf]

#179

Post by Luker88 »

fragmede wrote: post #45
throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

What is QUIC in your book? Given, say, $50 million of dev time, what would you go about fixing? And in what way?

IMHO QUIC is nice, but a disappointment, since it could have been so much more.

Does not handle unreliable messages, still only (multi)streaming, no direct support for multicast, 0-rtt which need a lot of stuff to be manually done TheRightWay or risk amplification attacks, the (imho) under-researched (and removed) forward error correction, and more.

I just restarted working on what I consider to be the solution to this, federated authentication and a bit more, but $50M is too far to be even a dream since I am not google.

Re: We need a replacement for TCP in the datacenter [pdf]

#180

Post by kortilla »

hinkley wrote: post #73
ggm wrote: post #4

Out of order delivery is fine in TCP within the window. It might be inefficient but it's not impossible, reassembly could be moved to userspace if userspace TCP was used. I have no problem with alternates to TCP in the DC with a crossbar fabric and far less loss, seems sensible. I wonder how it would play with QUIC and the session like behaviours now emerging.

> • In-order packet delivery This is a bit disingenuous, since it’s not the wire protocol but the kernel API that maintains the in order abstraction. With Jumbo packets you can still push a mountain of data without tripping up on “in order packet delivery” As developers we like this in order delivery to userspace because it vastly simplifies the code. We make up for the inefficiencies by processing dozens of hundreds…

Is it not considered a protocol violation to deliver out of order segments to the upper layer? That seems the same to me as abusing it to not require retransmits either.

Remember, middle boxes can fully adhere to the TCP standards and terminate your TCP connection and enforce ordering. If you notice that, you’re not really following the protocol, you’re just using its header format.

Post reply on HN