We need a replacement for TCP in the datacenter [pdf] • Hacker News Board

Re: We need a replacement for TCP in the datacenter [pdf]

#271

Post by wmf » Mon, Oct 31, 2022, 4:39 PM UTC

simplotek wrote: post #146

duped wrote: post #112

Earlier quoted context omitted.

Every major corporation has multiple research organizations doing nothing but invest in things that don't have immediate shareholder value. What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale. The scale at which these changes needs to happen to be practical are at the cluster level in a single data center. If you ca…

> What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale. I'd put it differently: it's paying up hundreds of billions in infrastructure to have some sort of gain. And which gain is that exactly? I see a lot of "the world is dumb but I am smart" comments in this thread but I saw no one presenting any clear advantage or…

It's figure 1 in the paper. Homa is over 10x faster than TCP (presumably CUBIC).

Re: We need a replacement for TCP in the datacenter [pdf]

#272

Post by AtNightWeCode » Mon, Oct 31, 2022, 4:43 PM UTC

friendzis wrote: post #164

throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols I think this is more of an artefact of horizontal scaling and port-contention. De-facto standard discovery mechanism DNS does not work with ports, so "well-known port" abstraction kinda fail…

I would say ports are mainly a problem on layers below transport even though some tech overuse ports.

Re: We need a replacement for TCP in the datacenter [pdf]

#273

Post by withinboredom » Mon, Oct 31, 2022, 4:43 PM UTC

sneak wrote: post #244

atoav wrote: post #126

Earlier quoted context omitted.

You mean like IPv6?

I think QUIC/http2 is a much better example. Google made that happen almost unilaterally via their Chrome dominance.

I mean, this is how new features come about, for the most part (look at ajax, from Microsoft's IE dominance). The consortium allows anyone to contribute, not just the dominate browser, but the dominate browser will always be able to experiment with new web features without having to discuss it with anyone.

Re: We need a replacement for TCP in the datacenter [pdf]

#274

Post by KaiserPro » Mon, Oct 31, 2022, 4:45 PM UTC

wmf wrote: post #266

KaiserPro wrote: post #206

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…

You're missing the fact that Stanford is the farm team for Google and Google is hyperscale. At scale, your "just spend more money" solutions are in fact more expensive than creating a new protocol. And like k8s, the new protocol can be sold to startups so they can "be like Google".

You're missing the point that maybe, just maybe, I'm part of a team that looks after >5 million servers.

You might also divine that while TCP can be a problem, a bigger problem is data affinity. Shuttling data from a next door rack costs less than one that's in the next door hall, and significantly less than the datacentre over. With each internal hop, the risk of congestion increases.

You might also divine that changing everything from TCP to a new, untested protocol across all services, with all that associated engineering effort, plus translation latency, might not be worth it. Especially as now all your observability and protocol routing tools don't work.

quick maths: a faster top of rack switch is possibly the same cost as 5 days engineering wage for a mid level google employee. How many new switches do you think you could buy with the engineering effort required to port everything to the new protocol, and have it stable and observable?

As a side note "oh but they are google" is not a selling point. Google has google problems half of which are things related to their performance/promotion system which penalises incremental changes in favour of $NEW_THING. HTTP2.0 was also a largely google effort designed to tackle latency over lossy network connections. which it fundamentally didn't do because a whole bunch of people didn't understand how TCP worked and were shocked to find out that mobile performance was shit.

Re: We need a replacement for TCP in the datacenter [pdf]

#275

Post by guenthert » Mon, Oct 31, 2022, 4:49 PM UTC

throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols.

Nobody forces them though. It would be much easier to publish a standard port number mapping than to develop a (or multiple) new protocols. Now you just need to motivate people to use it.

Re: We need a replacement for TCP in the datacenter [pdf]

#276

Post by wmf » Mon, Oct 31, 2022, 4:49 PM UTC

nickdothutton wrote: post #208

23 years ago I sat in a meeting with Sun, Intel, Mellanox, and 1 or 2 others. In that meeting we discussed putting an RDMA interface on individual hard drives, trays of RAM, CPUs, and other more exotic devices (like battery backed RAM, no conventional SSD in those days of course). You’d install RAM 1 42U rack at a time, disks likewise, CPUs in another rack and so on. All partitioned, controlled, managed, and of cours…

Disaggregation costs an absolute fortune. The network is 10% of the total datacenter cost and the network does not carry memory and PCIe traffic. If you make the network 10x faster to carry that, now it's 90% of the cost.

Re: We need a replacement for TCP in the datacenter [pdf]

#277

Post by wmf » Mon, Oct 31, 2022, 4:52 PM UTC

MuffinFlavored wrote: post #89

adobrawy wrote: post #48

Earlier quoted context omitted.

Homa protocl can be deployed on the basis of existing switches (not all of them, but some of them will play well). See also https://github.com/PlatformLab/HomaModule . In addition, a closed environment and the SDN's popularity in large data centers is a significant cost-reducing factors compared to typical IPv6 deployment.

Dumb question, there's no way to talk to a PC over the Internet with Homa, right? Since our home ISPs + routers are all only doing UDP/TCP over IPv4/IPv6? Homa is mainly for "LAN"?

Yes.

Re: We need a replacement for TCP in the datacenter [pdf]

#278

Post by TheRealDunkirk » Mon, Oct 31, 2022, 4:54 PM UTC

KaiserPro wrote: post #206

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…

God, I love it when the talk turns hyper-technical around here, and the Jedi masters turn up.

Re: We need a replacement for TCP in the datacenter [pdf]

#279

Post by wmf » Mon, Oct 31, 2022, 4:54 PM UTC

pantulis wrote: post #184

I am not really sure what Ousterhout means when talking "datacenter" but one of the key aspects of TCP networked applications is that it really doesn't matter if they are located in a datacenter, at the edge, in a mobile device, in a Raspberry at home or in your car and, better, they can be more or less moved from one hosting to another. Will this mean that application developers need to work with different network s…

Yes, you would have multiple protocols at some points. If you look at microservice architectures they already have an internal service mesh and external API gateways.

Re: We need a replacement for TCP in the datacenter [pdf]

#280

Post by donaldihunter » Mon, Oct 31, 2022, 5:01 PM UTC

JoeAltmaier wrote: post #263

delroth wrote: post #236

Earlier quoted context omitted.

> Google hasn't used TCP in the datacenter for years. That's absolutely false. I don't have any sources except for having worked at Google from 2013-2022, but it's not like you quoted any sources either, so... There's a reason why Google is still releasing stuff like TCP BBR (2017).

Well, the first link googling 'google datacenter hardware' is google's article on how they don't use standard TCP hardware or software in their datacenters. But I guess that was too much to ask...

There's a big difference between "hasn't used TCP in the datacenter for years" and "don't use standard TCP hardware or software in their datacenters". Google uses TCP with non-standard configuration, but they still use TCP.