We need a replacement for TCP in the datacenter [pdf] • Hacker News Board

Re: We need a replacement for TCP in the datacenter [pdf]

#231

Post by Bluecobra » Mon, Oct 31, 2022, 1:10 PM UTC

adamcharnock wrote: post #168

tezza wrote: post #155

Earlier quoted context omitted.

Do you have any specific problems you can elaborate with the UDP ? UDP used successfully many places.

I’m really looking forwarding to seeing the original commenters reply on this. But I’ll share my experience too. I’ve found UDP to be great for latency but pretty awful for throughout. Especially over longer routes (ie inter-region transports). Also, if you fire UDP packets out of a machine in a tight loop then there is every chance you could overload various buffers and just loose them (depending on the networking h…

> I’ve found UDP to be great for latency but pretty awful for throughout.

UDP/multicast can provide excellent throughput. It's the de facto standard for market data on all major financial exchanges. For example, the OPRA feed (which is a consolidated market data feed of all options trading) can easily burst to ~17Gbps. Typically there is a "A" feed and a "B" feed for redundancy. Now you're talking about ~34Gbps of data entering your network for this particular feed.

Also, when network engineers do stress testing with iperf we typically use UDP to avoid issues with TCP/overhead.

Re: We need a replacement for TCP in the datacenter [pdf]

#232

Post by zrail » Mon, Oct 31, 2022, 1:11 PM UTC

GTP wrote: post #175

idlehand wrote: post #139

Earlier quoted context omitted.

Never thought about that before. Ethernet supports extremely high levels of data transmission. ISB C for intrgrated charging ND data transfer makes sense, but why are there HDMI cables?

I think it would be an overkill to use a networking protocol to connect exactly two devices. Plus if you have something specific to video stream transfer you could maybe do some optimization specific to that use case, although I can't think of any at the moment.

Excepting HDMI the parents examples are all networks with more than one peer. Thunderbolt and USB3 can both have arbitrary trees of nodes.

Re: We need a replacement for TCP in the datacenter [pdf]

#233

Post by Bluecobra » Mon, Oct 31, 2022, 1:16 PM UTC

nextstepguy wrote: post #204

lamontcg wrote: post #131

Earlier quoted context omitted.

After having lived through Amazon's early (pre-2003ish) UDP-based networking I got a laugh around 2006-ish or so reading about how facebook was into UDP. I assume there are people who worked there who still have the scars.

Multicast storms happened regularly back in 2004

True, there were tons of crappy hardware still in production at that time. The first job I had out of college consisted of crappy 3Com hubs (not switches) so something like Norton Ghost could take down the whole network since multicast would get flooded everywhere. Nowadays this is a less of a problem as hubs are long gone and most switches have IGMP snooping by default and would only forward mutlicast frames that someone wants.

A bad client can still cause problems though, like sending a high rate of multicast packets with a TTL of 1.

Re: We need a replacement for TCP in the datacenter [pdf]

#234

Post by HippoBaro » Mon, Oct 31, 2022, 1:31 PM UTC

I think that was an interesting read. I have worked on an userspace implementation of TCP via DPDK so I have sympathy for the limitations mentioned (the load balancing and thread scheduling argument are very accurate).

However I would have liked a section dedicated as to why hardware accelerated UDP wouldn’t be an adequate solution vs a whole new protocol. It seems to me it provides a solid basis for achieving the results the author wants to bring about?

Re: We need a replacement for TCP in the datacenter [pdf]

#235

Post by bayindirh » Mon, Oct 31, 2022, 1:34 PM UTC

amluto wrote: post #212

KaiserPro wrote: post #206

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…

The paper explicitly addresses Infiniband.

IB does not work TCP/IP by default. You can either run TCP over IB, which has a performance penalty, or you can directly run in Ethernet mode, which is something completely different.

Re: We need a replacement for TCP in the datacenter [pdf]

#236

Post by delroth » Mon, Oct 31, 2022, 1:51 PM UTC

JoeAltmaier wrote: post #227

Google hasn't used TCP in the datacenter for years. What they use I don't know. But it's even custom switches with custom chips. My son did work in graduate school for a clean-slate network implementation of a network for the datacenter. Maybe Google, I don't remember. One issue I remember they addressed was, scheduling bandwidth for VM migration within their datacenter cloud. See, some customer reserves a 'machine'…

> Google hasn't used TCP in the datacenter for years.

That's absolutely false. I don't have any sources except for having worked at Google from 2013-2022, but it's not like you quoted any sources either, so...

There's a reason why Google is still releasing stuff like TCP BBR (2017).

Re: We need a replacement for TCP in the datacenter [pdf]

#237

Post by fulafel » Mon, Oct 31, 2022, 1:56 PM UTC

bheadmaster wrote: post #172

bruce343434 wrote: post #153

Earlier quoted context omitted.

What’s the issue with fitting messages in streams?

One of the issues I can think of is head-of-line blocking [0]. If you're sending messages of different priorities over the same channel, an error in sending a low-priority message, high-priority messages will have to wait until the low-priority message is properly re-transmitted. [0] https://en.wikipedia.org/wiki/Head-of-line_blocking https://en.wikipedia.org/wiki/Head-of-line_blocking

Yes, but this need happen only when your data center network is congested, which is hopefully rare and is relatively cheap to fix by adding capacity. And in congestion cases you need TCP's back off ability.

Getting rid of head-of-line blocking also makes messages happen in order. Making messages (sometimes) happen out of order would drastically increase implementation complexity for a lot of apps.

Re: We need a replacement for TCP in the datacenter [pdf]

#238

Post by pclmulqdq » Mon, Oct 31, 2022, 2:00 PM UTC

samgaw wrote: post #183

alexgartrell wrote: post #84

Earlier quoted context omitted.

> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. This is ridiculous. Hyperscalars see an immediate ROI from efficiency/reliability improvements and actively invest in TCP alternatives all of the time. It's just really hard. Networking companies see an ability to differentiate their products from their peers and work on this kind of thing as well…

> FWIW, I wouldn't be surprised if the high-performance RDMA networks being put together for AI workloads were the thing that grew into the "next" thing. Maybe we were just early in giving (HFT) customers RDMA back in ~2007[1][2] but I don't see it entering the mainstream anytime soon. And after a relatively short 20 years of adoption, the "next" thing for hyperscalers is not going to be the next thing for everyone e…

HFT networks are also a lot smaller than hyperscaler datacenters, and designed with more cross-sectional bandwidth. A good chunk of the traffic (trading-related messages) also tends to not use congestion control.

In large web company datacenters, RDMA and RoCE have had a much "rockier" path forward.

Re: We need a replacement for TCP in the datacenter [pdf]

#239

Post by bewo001 » Mon, Oct 31, 2022, 2:12 PM UTC

patrec wrote: post #170

friendzis wrote: post #164

Earlier quoted context omitted.

> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols I think this is more of an artefact of horizontal scaling and port-contention. De-facto standard discovery mechanism DNS does not work with ports, so "well-known port" abstraction kinda fail…

> De-facto standard discovery mechanism DNS does not work with ports Yes, it does, see SRV records.

So true, but how many developers know about them? The API situation does not help either.

Re: We need a replacement for TCP in the datacenter [pdf]

#240

Post by simplotek » Mon, Oct 31, 2022, 2:13 PM UTC

fragmede wrote: post #161

simplotek wrote: post #146

Earlier quoted context omitted.

> What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale. I'd put it differently: it's paying up hundreds of billions in infrastructure to have some sort of gain. And which gain is that exactly? I see a lot of "the world is dumb but I am smart" comments in this thread but I saw no one presenting any clear advantage or…

I mean the goal is more performance, especially if you can get more performance out of the same hardware. Faster setup times, faster connections, more connections, maybe faster teardown. Lower contention on saturated links. Inside of the datacenter is a controlled environment where something like that could work. Replacing TCP over the Internet at large is going to be an uphill battle. Still, if we're replacing the w…

> I mean the goal is more performance, especially if you can get more performance out of the same hardware.

Are there actual numbers demonstrating this?

I mean, people are advocating wasting billions revamping infrastructure. What kind of performance are you hoping to buy with those billions? And are those gains worth it, or is just sake for the sake of change?

Sometimes things are indeed good enough.