Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a replacement for TCP in the datacenter [pdf]

web.stanford.edu

Post reply on HN
Print view View original

291–300 of 329 posts

Re: We need a replacement for TCP in the datacenter [pdf]

#291

Post by oconnor663 »

alexgartrell wrote: post #127
oconnor663 wrote: post #87

Earlier quoted context omitted.

> 50 years of clock time and thousands of years of engineer time It's not just the size of the investment, it's that it's the protocol everyone uses to talk to other people's machines, and you can't upgrade or replace other people's machines.

In this case we're talking about within the Datacenter, and you could conceivably update every network device and system to talk the new thing if you wanted. This is more achievable at a hyperscalar, where there tends to be TCP gives you three things: 1. Reasonable performance - This is hard but not impossible to replicate 2. Reliability - This is very hard to replicate because networking edge cases are very hard to…

> In this case we're talking about within the Datacenter

Oh gotcha. It's right there in the title, but missed it somehow :p

Re: We need a replacement for TCP in the datacenter [pdf]

#292

Post by astrobe_ »

cryptonector wrote: post #250
amluto wrote: post #219

Earlier quoted context omitted.

Once chunked encoding is in the picture, even HTTP/1.1 sends messages, not streams, under the hood.

Correct.

I don't think so, or we disagree on the meaning of words. Chunking is not message-oriented in the same sense that UDP is.

With chunking, you basically just insert markers into the stream; this does not imply by any mean that the stream has been split in multiple messages - as a matter of this is taken care of by the lower levels of the client/server, the middle/higher levels certainly don't want to deal with it. This only is a perverted solution to the problem of dynamically generated "messages" (mainly HTML pages), that has been further perverted to implement gruesomely message-oriented "protocols" (Comet and others, IIRC).

UDP, on the other hand is based on datagrams. They can be split into smaller packets on the wire but they are reassembled at the network stack level so no program can even see it happened unless they insist on it.

Websocket is much closer to a message oriented protocol over a streaming pipe than HTTP chunking is.

Re: We need a replacement for TCP in the datacenter [pdf]

#293

Post by fragmede »

ksec wrote: post #267
wmf wrote: post #266

Earlier quoted context omitted.

You're missing the fact that Stanford is the farm team for Google and Google is hyperscale. At scale, your "just spend more money" solutions are in fact more expensive than creating a new protocol. And like k8s, the new protocol can be sold to startups so they can "be like Google".

AWS is True HyperScale. Even more so than Google. And yet their spend more money solution on hardware seems to work fine.

Do we know for a fact that AWS does or doesn't use TCP on their backend? https://news.ycombinator.com/item?id=33402364 leads me to believe Google doesn't.

Re: We need a replacement for TCP in the datacenter [pdf]

#294

Post by kazen44 »

Bluecobra wrote: post #231
adamcharnock wrote: post #168

Earlier quoted context omitted.

I’m really looking forwarding to seeing the original commenters reply on this. But I’ll share my experience too. I’ve found UDP to be great for latency but pretty awful for throughout. Especially over longer routes (ie inter-region transports). Also, if you fire UDP packets out of a machine in a tight loop then there is every chance you could overload various buffers and just loose them (depending on the networking h…

> I’ve found UDP to be great for latency but pretty awful for throughout. UDP/multicast can provide excellent throughput. It's the de facto standard for market data on all major financial exchanges. For example, the OPRA feed (which is a consolidated market data feed of all options trading) can easily burst to ~17Gbps. Typically there is a "A" feed and a "B" feed for redundancy. Now you're talking about ~34Gbps of da…

Multicast throughput is hard to measure because it is... well, multicast.

Depending on where your RP's are, and how you are transmitting multicast packets across a core, multicast performance can vary a lot.

The main advantage of multicast however, is that throughput between RP's doesn't need to be very large..

Re: We need a replacement for TCP in the datacenter [pdf]

#295

Post by cryptonector »

astrobe_ wrote: post #292
cryptonector wrote: post #250

Earlier quoted context omitted.

Correct.

I don't think so, or we disagree on the meaning of words. Chunking is not message-oriented in the same sense that UDP is. With chunking, you basically just insert markers into the stream; this does not imply by any mean that the stream has been split in multiple messages - as a matter of this is taken care of by the lower levels of the client/server, the middle/higher levels certainly don't want to deal with it. This…

> I don't think so, or we disagree on the meaning of words.

Well, when there's four people in the conversation, that happens.

u/josephg's complaint at the top of this thread is that people use TCP but still have to add framing in their application protocols. u/josephg said something to the effect of how few protocols do no framing and mentioned HTTP/1, but even HTTP/1.1 w/ chunked transfer encoding adds framing, and even HTTP/1.0 w/ definite content-length also has framing (CRLFs) for the request headers themselves, and effectively frames bodies with CRLF at the start and EOF at the end.

Chunks are definitely not datagrams, just as TLS records aren't either, and just as TCP segments aren't either. But they have framing, which u/josephg complained about.

Framing of some sort is unavoidable. My point, besides the inevitability of application-layer framing, is that a datagram- or message-oriented transport won't make things trivial for apps anymore than TCP did.

Re: We need a replacement for TCP in the datacenter [pdf]

#296

Post by dragontamer »

Hmm, haven't read the paper yet, but I immediately did "Ctrl+F sctp" and didn't find anything.

I know that sctp was the next generation stream-oriented protocol designed to fix the out-of-order message problem in commnications, as well as a whole bunch of connection issues (4-way handshake instead of 3-way for better open/close. Datagram oriented in-order stream, so that every packet has a proper size involved. Etc. etc.)

As far as I know, sctp should solve all the requirements in section 2 of this paper (except "load balancing", which might be solved by lower-level protocol sharing of some kind?). So its weird to not see sctp discussed.

--------

Yeah, sctp ain't popular, but these exact sets of problems / requirements and issues with TCP have been known for decades. SCTP, is also a decades-old protocol (though not as old as TCP), and is the most obvious solution to the problem (and already supported by Linux).

Re: We need a replacement for TCP in the datacenter [pdf]

#297

Post by primeblue »

TheRealDunkirk wrote: post #278
KaiserPro wrote: post #206

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…

God, I love it when the talk turns hyper-technical around here, and the Jedi masters turn up.

No post body was provided.

Re: We need a replacement for TCP in the datacenter [pdf]

#298

Post by KaiserPro »

counttheforks wrote: post #282
KaiserPro wrote: post #206

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…

> Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) So storage doesn't use TCP, except for the protocols that are actually used, which do use TCP?

Depends on what you are using, for connecting block stores, you'll use some sort of fabric. That is Fibre channel, SAS, NVME over something or other

If you are using GPFS, then you can do stuff over IB, but I don't know how that works. Lustre I imagine does lustre things over RDMA.

For everything else, NFS all the things. pNFS means that you can just throw servers at the problem and let the network figure it out.

But again, if IO speed is critical, you move IO over to a dedicated fabric of somesort. for most thing NFS is good enough. (except databases, its possible but not great. but then depending on your docker setup, you might be kneecaping your performance because overlayfs is causing io amplification)

Re: We need a replacement for TCP in the datacenter [pdf]

#299

Post by bsder »

dooglius wrote: post #207
bsder wrote: post #109

Earlier quoted context omitted.

> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. And yet every time hardware designers get the chance they redesign Ethernet and IPv4--poorly. See: HDMI 2.0+, USB 3.0+, Thunderbolt 3.0+, etc. My suspicion is that this paper works fine beween pairs of peers and immediately goes straight to hell after that. It is extremely suspicious that there is…

Why do you say that these protocols are worse than Ethernet/IPv4? I'm not intimately familiar with any at L2/L3, but I don't think any have hacks as bad as ARP. (USB does have some weirdness at L1 though I know.)

> Why do you say that these protocols are worse than Ethernet/IPv4?

Here's an example: I connected my nice expensive audio interface to my Thunderbolt port. It worked great! Then I moved a window on my monitor and all hell broke loose. In spite of the fact that it had way more than enough bandwidth to handle everything.

See, Thunderbolt doesn't have the ability to say "This tiny packet going to there needs priority and you need to break up those giant display packets."

Ethernet has solved problems like these in standards. They're not always implemented on particular chipsets, but they exist, and you generally can buy a product that has them.

Everything Ethernet has done and standardized has generally been for a reason. If you don't implement Ethernet, then you are starting over from scratch and will have to reimplement all of that stuff.

And you're probably not smarter than the guys who did it for Ethernet.

(If I'm being charitable: what's happened is that lot of standards tried to be more cost optimized than Ethernet. The problem is that transistor prices keep coming down. Eventually the price delta between Ethernet and becomes inconsequential and you're basically left with real Ethernet and "kinda crappy" Ethernet at almost the same price.)

Re: We need a replacement for TCP in the datacenter [pdf]

#300

Post by adamcharnock »

Bluecobra wrote: post #231
adamcharnock wrote: post #168

Earlier quoted context omitted.

I’m really looking forwarding to seeing the original commenters reply on this. But I’ll share my experience too. I’ve found UDP to be great for latency but pretty awful for throughout. Especially over longer routes (ie inter-region transports). Also, if you fire UDP packets out of a machine in a tight loop then there is every chance you could overload various buffers and just loose them (depending on the networking h…

> I’ve found UDP to be great for latency but pretty awful for throughout. UDP/multicast can provide excellent throughput. It's the de facto standard for market data on all major financial exchanges. For example, the OPRA feed (which is a consolidated market data feed of all options trading) can easily burst to ~17Gbps. Typically there is a "A" feed and a "B" feed for redundancy. Now you're talking about ~34Gbps of da…

That’s interesting. And I’m sure they have some very knowledgable people working for them who may(/will) know things I don’t.

That being said, it wouldn’t surprise me if they were pushing 17G of UDP on 100G transports. Probably with some pretty high-end/expensive network hardware with huge buffers. I.e you can do it if you’ve got the money, but I bet TCP would still have better raw throughput.

Post reply on HN