Could we get the title replaced to match the document, rather than the cliched, bombastic "It's time to ..." formulation?
Ousterhout actually wrote:
> We Need a Replacement for TCP in the Datacenter
We need a replacement for TCP in the datacenter [pdf]
81–90 of 329 posts
Re: We need a replacement for TCP in the datacenter [pdf]
#82Out of order delivery is fine in TCP within the window. It might be inefficient but it's not impossible, reassembly could be moved to userspace if userspace TCP was used. I have no problem with alternates to TCP in the DC with a crossbar fabric and far less loss, seems sensible. I wonder how it would play with QUIC and the session like behaviours now emerging.
There was a time when out of order packets triggered congestion handling in TCP stacks which drastically reduces performance. This is where the concern comes from. I think it's a bit out dated though, I think the newer schedulers ignore out of order delivery. I've also seen problems on some embedded stacks, but that could easily be argued that the implementation is wrong. But I've seen things like credit card termina…
I can't fault Ousterhout for writing in support of a new(ish) idea but his language here went to "forbidden" when in fact it's just "strongly disliked"
TCP the protocol knows how to re-assemble out of order. What I think he's doing is making it a higher task to do it, outside of the protocol, or else providing some mechanism in user process space, amenable to threading.
I can believe an async model of "tell me when this is complete" would work well with a bitmap/bloom filter type gate on what "has to be complete" to proceed.
I like his writing. I was a fan of tcl/tk and used expect heavily back in the past.
Re: We need a replacement for TCP in the datacenter [pdf]
#83Earlier quoted context omitted.
What is QUIC in your book? Given, say, $50 million of dev time, what would you go about fixing? And in what way?
In addition to QUIC, KCP [1] is another reliable low-latency protocol that sits on top of UDP that might be interesting. And unlike RFC 9000/9001 (QUIC), encryption is optional. I haven't really seen it mentioned much outside of primarily China-focused projects, like V2Ray [2], but there is also some English information in their Git repo [2]. [1]: https://github.com/skywind3000/kcp > [2]: https://www.v2fly.org/en_US/…
KCP uses a brute force congestion control algorithm that is unfair and inefficient. It is also poorly specified, which is probably why it is less commonly used outside circumvention circles.
Re: We need a replacement for TCP in the datacenter [pdf]
#84Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…
> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo.
This is ridiculous.
Hyperscalars see an immediate ROI from efficiency/reliability improvements and actively invest in TCP alternatives all of the time. It's just really hard.
Networking companies see an ability to differentiate their products from their peers and work on this kind of thing as well. I did a 3 second google for "QUIC acceleration Mellanox" and got a hit on Nvidia's blog right away.
You just can't trivially replace something with an investment totally 50 years of clock time and thousands of years of engineer time. It will either take a long time or a massive shift in needs/technology. FWIW, I wouldn't be surprised if the high-performance RDMA networks being put together for AI workloads were the thing that grew into the "next" thing.
Re: We need a replacement for TCP in the datacenter [pdf]
#85> Although Homa is not API-compatible with TCP, it should be possible to bring it into widespread usage by integrating it with RPC frameworks.
Not being sockets API compatible kinda sucks. Ok, we could use a new connect() variation that allows for earlier data send, but the API being mostly similar would help -- there's a ton of socket code out there!
As for RPC, well, RPC is mostly a thing of the past with most everything today is HTTPS, but usually through libraries that do the I/O, so it's possible to retrofit a new transport protocol into them. But why not just use QUIC within the datacenter?
Re: We need a replacement for TCP in the datacenter [pdf]
#86I wrote a blog post that might be interesting to those wanting an introduction into some of the basics of the problems called "02-FEB-2011: Why is there packet loss ?" [0]
Re: We need a replacement for TCP in the datacenter [pdf]
#87Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…
> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. This is ridiculous. Hyperscalars see an immediate ROI from efficiency/reliability improvements and actively invest in TCP alternatives all of the time. It's just really hard. Networking companies see an ability to differentiate their products from their peers and work on this kind of thing as well…
> 50 years of clock time and thousands of years of engineer time
It's not just the size of the investment, it's that it's the protocol everyone uses to talk to other people's machines, and you can't upgrade or replace other people's machines.
Re: We need a replacement for TCP in the datacenter [pdf]
#88Earlier quoted context omitted.
Forgive my ignorance but why isn't SCTP more frequently used in DCs? I know it misbehaves with home routers etc. but shouldn't be a factor here.
I suspect there are a couple of contributors. TCP is prevalent on the internet, so you need a fairly strong motivation and benefits to adopt a second protocol. A lot of engineering also doesn't get underlying networking, so one of the successes of TCP is it's a file descriptor that you either write to or read from and magic makes it come out the other side. I've seen tech leadership on networking centric products kno…
How does your theory that the failure of SCTP is because a) people don’t understand networking and b) tcp eats up all the development oxygen explain QUIC?
I’m also not sure what you mean but DCs within a major cloud provider are majority AFAIK running truly isolated networks interconnected directly with fiber.
If you haven’t yet, I would recommend reading the very original QUIC paper. It was extremely astute and showed quite a deep understanding of what the problems were with TCP done by network engineers who really knew their shit (I got to interact with some of them when I was at Google). They talk about the failures of SCTP on technical levels and non-technical headwinds that weren’t accounted for like ossification. To my knowledge QUIC is SCTP 2.0 - it provides much of the same features and in a way that could actually leave the lab.
Re: We need a replacement for TCP in the datacenter [pdf]
#89Well, the key would be to develop and deploy Homa in a DC and test in implementation at scale. If it actually ameliorates the perceived shortcomings of TCP that make nothing in TCP worth keeping as this author says, then cool. My only complain with issues like this is the cost of implementation. Someone has to pay to build a DC around it or increase the cost of maintenance for several years to a decade while supporti…
Homa protocl can be deployed on the basis of existing switches (not all of them, but some of them will play well). See also https://github.com/PlatformLab/HomaModule . In addition, a closed environment and the SDN's popularity in large data centers is a significant cost-reducing factors compared to typical IPv6 deployment.
Dumb question, there's no way to talk to a PC over the Internet with Homa, right? Since our home ISPs + routers are all only doing UDP/TCP over IPv4/IPv6? Homa is mainly for "LAN"?