Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a replacement for TCP in the datacenter [pdf]

web.stanford.edu

Post reply on HN
Print view View original

241–250 of 329 posts

Re: We need a replacement for TCP in the datacenter [pdf]

#241

Post by KaiserPro »

pclmulqdq wrote: post #228
KaiserPro wrote: post #206

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…

The reality of today's large datacenters is that almost all of them have almost all of their traffic on TCP unless the owners of the datacenter have made a conscious effort to not use TCP. The highest-traffic applications, usually databases and storage systems, pretty much all use TCP unless you are buying a purpose-built HPC scale-out storage system (like a Lustre cluster). Most people who build a datacenter today u…

Fibre channel is dead, long live fibre channel.

I agree a lot of things are on TCP, but I don't think its a massive problem, unless you are running close to the limit of your core network. And one solution to that is to upgrade your core network....

Failing that, implementing some load balancing/partitioning systems to make sure data-processing affinity is best matched. This the better solution, because it yields other advantages as well. But its not the easiest, unless you have a good scheduler

Re: We need a replacement for TCP in the datacenter [pdf]

#242

Post by LinuxBender »

I believe something missed in these discussions is that pertaining to "the data-center". A data-center is not a technology, it is a grouping of assets. Those assets and their associated services need to communicate not just with each other, but with other assets and services on the internet.

Regardless of what incredible technical solution one creates, it will have to allow for simultaneous existence of current IP protocols in parallel with whatever proposed replacements to exist seamlessly with one another or significant adoption would never occur. A data-center is not an isolated bubble, at least not any more unless one wants to translate said protocols through a single point of success gateway. Should such a replacement ever occur it will have to be done piece by piece until there is nothing left using current IP protocols.

So I believe people should all create their proposed protocols and give businesses a low-friction path to adoption one service at a time. As more applications adopt said protocols, the most popular, highest reliable, most performant least friction path will likely win and if successful then at some distant point in the future perhaps most existing IP protocols could be deprecated. As a reminder, each application will have to adopt libraries to speak on this protocol and know how to utilize it. There will be a "battle hardening" period to work out the bugs and security controls. All of the network gear in the entire path between data-centers and clients will need OS/firmware/Asic updates to understand this protocol. Given the transition speed to IPv6 as an example, this could be a very long road.

There is also some discussion of QUIC and SPDY. Those are not new IP protocols. Those are new standards within an existing L7 application protocol HTTP that still utilize existing L3/L4 protocols. Replacing TCP and UDP? in the data-center means a new protocol in /etc/protocols not encapsulated in an existing protol such as protol 6 TCP or protocol 17 UDP. The network gear and OS on every device in the path will need to understand this new protocol. Tools such as tcpdump and libraries such as libpcap would need to be updated to understand these new protocols before they would even be used in a development environment.

Could it be that I misunderstood the intent and perhaps we just want yet another new L7 application protocol on top of UDP?

Re: We need a replacement for TCP in the datacenter [pdf]

#243

Post by [deleted] »

emn13 wrote: post #189
simplotek wrote: post #142

Earlier quoted context omitted.

> Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. If this was true then how do you explain that the likes of AWS, the same company w…

It's not obvious to me that replacing TCP really is harder than designing your "own" chip. Scarequotes here because those graviton chips (that's what you're referring to, I think?) are of course ARM chips, so they're not designing something fresh; they're adapting a very mature design to their own needs. In terms of interoperability, a custom chip based on a standard design is probably a simpler, more locally address…

[deleted]

Re: We need a replacement for TCP in the datacenter [pdf]

#244

Post by sneak »

atoav wrote: post #126
kebman wrote: post #110

Earlier quoted context omitted.

Yes you can. Just offer a better product, and people will buy it instead of the old or bad product. Better yet, make the new product backwards compatible, and fewer people will have qualms about forking out for it. Better yet, do an aggressive takeover, like Microsoft did, and just force the entire industry to adopt your stuff...

You mean like IPv6?

I think QUIC/http2 is a much better example.

Google made that happen almost unilaterally via their Chrome dominance.

Re: We need a replacement for TCP in the datacenter [pdf]

#245

Post by bewo001 »

kevin_nisbet wrote: post #59
ggm wrote: post #4

Out of order delivery is fine in TCP within the window. It might be inefficient but it's not impossible, reassembly could be moved to userspace if userspace TCP was used. I have no problem with alternates to TCP in the DC with a crossbar fabric and far less loss, seems sensible. I wonder how it would play with QUIC and the session like behaviours now emerging.

There was a time when out of order packets triggered congestion handling in TCP stacks which drastically reduces performance. This is where the concern comes from. I think it's a bit out dated though, I think the newer schedulers ignore out of order delivery. I've also seen problems on some embedded stacks, but that could easily be argued that the implementation is wrong. But I've seen things like credit card termina…

IBM AIX's TCP can either do selective ACKs or handle out-of-order packets. Which we were told when a firewall started reordering packets. It simply dropped out of order packets and therefore triggered congestion handling.

Re: We need a replacement for TCP in the datacenter [pdf]

#246

Post by xxpor »

simplotek wrote: post #142
throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

> Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. If this was true then how do you explain that the likes of AWS, the same company w…

https://aws.amazon.com/blogs/hpc/in-the-search-for-performan...

Re: We need a replacement for TCP in the datacenter [pdf]

#247

Post by jmull »

I get the issue with TCP, but I'm not sure about Homa... e.g., why not UDP with some homa-like semantics on top? That might really ease the "Getting there from here" issue.

In fact, what's actually being suggested is for applications to replace calls to TCP-based APIs with calls to gRPC APIs (or other high-level RCP APIs), where the transport layer becomes an implementation detail. Fair enough, but this is a very round-about way to go about it.

Re: We need a replacement for TCP in the datacenter [pdf]

#248

Post by EricE »

gjulianm wrote: post #174
throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

> We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use huma…

"The problem with networks is that they're very complex systems, with multiple elements along the way, made by different manufacturers, often with different owners" Ah, how people forget the early days of networking. I remember vividly the early days of the Networld/Interop trade show - Interop was in the name because if, as a vendor, your equipment couldn't integrate with the show network they would throw your booth off the show floor.

That's how bad interoperability in the early days was!

Re: We need a replacement for TCP in the datacenter [pdf]

#249

Post by cryptonector »

josephg wrote: post #158
cryptonector wrote: post #128

Earlier quoted context omitted.

> The only kinda-sorta exceptions I know about are HTTP/1.1 and telnet. But HTTP/1.1 is still a message oriented protocol; just with file-sized messages. (And even this stops being true with http2 anyway). No, HTTP/2 and QUIC do not change the semantics of HTTP. Also, you can have endless streams with HTTP/1.1: just use chunked encoding to POST/PUT and use Range: bytes=0- and chunked encoding for GET and chunked enco…

Correct me if I’m wrong, but doesn’t h2 still break up requests and responses into smaller message frames in order to do multiplexing? Those message frames are what I’m talking about - as I understand it, they are, yet again, a message oriented protocol layered on top of tcp.

Yes, and even HTTP/1.1 does that. I forget if telnet does something similar, but I suspect it must because it can send control data. The FTP protocol, and the BSD r-command might be the only ones that truly do no additional framing (FTP for data connections, r-commands post-login).

Re: We need a replacement for TCP in the datacenter [pdf]

#250

Post by cryptonector »

amluto wrote: post #219
cryptonector wrote: post #128

Earlier quoted context omitted.

> The only kinda-sorta exceptions I know about are HTTP/1.1 and telnet. But HTTP/1.1 is still a message oriented protocol; just with file-sized messages. (And even this stops being true with http2 anyway). No, HTTP/2 and QUIC do not change the semantics of HTTP. Also, you can have endless streams with HTTP/1.1: just use chunked encoding to POST/PUT and use Range: bytes=0- and chunked encoding for GET and chunked enco…

Once chunked encoding is in the picture, even HTTP/1.1 sends messages, not streams, under the hood.

Correct.
Post reply on HN