Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a replacement for TCP in the datacenter [pdf]

web.stanford.edu

Post reply on HN
Print view View original

211–220 of 329 posts

Re: We need a replacement for TCP in the datacenter [pdf]

#211

Post by jstimpfle »

adamcharnock wrote: post #168
tezza wrote: post #155

Earlier quoted context omitted.

Do you have any specific problems you can elaborate with the UDP ? UDP used successfully many places.

I’m really looking forwarding to seeing the original commenters reply on this. But I’ll share my experience too. I’ve found UDP to be great for latency but pretty awful for throughout. Especially over longer routes (ie inter-region transports). Also, if you fire UDP packets out of a machine in a tight loop then there is every chance you could overload various buffers and just loose them (depending on the networking h…

UDP is little more than IP, so there isn't a technical reason why UDP couldn't be just as fast as TCP _per se_. But from when I was toying with writing a stream abstraction on top of UDP in Linux userspace, I came to the same conclusion, it's hard to achieve high throughput.

My guess is that this is in part because achieving high throughput on IP is hard and in part because it's never going to be super efficient at this level (in userspace, on top of kernel infrastructure that might not be as optimized towards throughput like it is in the case of TCP).

Re: We need a replacement for TCP in the datacenter [pdf]

#212

Post by amluto »

KaiserPro wrote: post #206

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…

The paper explicitly addresses Infiniband.

Re: We need a replacement for TCP in the datacenter [pdf]

#213

Post by Phelinofist »

uluyol wrote: post #100
vlovich123 wrote: post #92

Earlier quoted context omitted.

I’m not aware of them using something other than TCP internally (I’m sure by now they’ve migrated to QUIC but I’m not sure that QUIC necessarily solves some of the scaling challenges / optimizes for gRPC and low latency).

Google is using remote memory accesses rather than TCP for at least some classes of traffic (e.g. a caching system). They've been publishing details about how it all works too. Also, they have a transport (Pony express) developed specifically for RPCs, rather than byte streams or datagrams. Links: https://research.google/pubs/pub51341/ , https://research.google/pubs/pub50590/ , https://research.google/pubs/pub48630/…

Can someone ELI5 how remote memory access works?

Re: We need a replacement for TCP in the datacenter [pdf]

#214

Post by arka2147483647 »

throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

> It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols. And then there's all the non-backend problems!

The paper argues the in '3.1 Stream orientation' section, that stream orientation is a problem for TCP, and says that most apps send messages instead, and the better protocol should handle messages, natively, etc. Which is a good point I think.

But back to TCP. What do you do, if you need to send Messages between applications in TCP? Preferably those Messages would be encrypted also.

You could make up your own protocol, but you probably would rather not! So you use something that is readily available, and does messages, encryption, etc. Would be nice if there were also a ready to use load balancers, caches, tools to debug it, etc

Now, what would be such a protocol.

Why HTTPS, of course.

So I kind of think that the lack of a low level Message Protocol has lead us, as an industry, to coalesce these features bit-by-bit on top of HTTP. It's not perfect by any means, but it does the job.

Re: We need a replacement for TCP in the datacenter [pdf]

#215

Post by lizknope »

For those unfamiliar with the author.

https://en.wikipedia.org/wiki/John_Ousterhout

He is probably most famous for having created the Tcl language and Tk GUI library. He also worked on the Sprite distributed operating system, the Magic VLSI design tool, and a bunch of other things.

Re: We need a replacement for TCP in the datacenter [pdf]

#216

Post by KaiserPro »

amluto wrote: post #212
KaiserPro wrote: post #206

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…

The paper explicitly addresses Infiniband.

not really. they conflate infiniband with RoCE which given they have different semantics on congestion control, I'd say is a bit of a whoopsey.

if they are using RoCE, are they using DCB to avoid loss(well make it "lossless")? the paper implies otherwise.

Re: We need a replacement for TCP in the datacenter [pdf]

#217

Post by amluto »

dooglius wrote: post #207
bsder wrote: post #109

Earlier quoted context omitted.

> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. And yet every time hardware designers get the chance they redesign Ethernet and IPv4--poorly. See: HDMI 2.0+, USB 3.0+, Thunderbolt 3.0+, etc. My suspicion is that this paper works fine beween pairs of peers and immediately goes straight to hell after that. It is extremely suspicious that there is…

Why do you say that these protocols are worse than Ethernet/IPv4? I'm not intimately familiar with any at L2/L3, but I don't think any have hacks as bad as ARP. (USB does have some weirdness at L1 though I know.)

I’ve generally considered IPv6 neighbor discovery to be a worse hack than ARP. ARP is a straightforward, fairly clean hack to layer the IPv4 addressing scheme over Ethernet, and it doesn’t pollute IPv4 itself. Neighbor discovery layers IPv6 on top of pseudo-IPv6, where the latter operates without knowledge of MAC addresses but nonetheless hardcodes knowledge of Ethernet. But hey, it eliminated the use of Ethernet broadcast in favor of a more complex but functionally identical multicast scheme.

Re: We need a replacement for TCP in the datacenter [pdf]

#218

Post by stevewatson301 »

jstimpfle wrote: post #211
adamcharnock wrote: post #168

Earlier quoted context omitted.

I’m really looking forwarding to seeing the original commenters reply on this. But I’ll share my experience too. I’ve found UDP to be great for latency but pretty awful for throughout. Especially over longer routes (ie inter-region transports). Also, if you fire UDP packets out of a machine in a tight loop then there is every chance you could overload various buffers and just loose them (depending on the networking h…

UDP is little more than IP, so there isn't a technical reason why UDP couldn't be just as fast as TCP _per se_. But from when I was toying with writing a stream abstraction on top of UDP in Linux userspace, I came to the same conclusion, it's hard to achieve high throughput. My guess is that this is in part because achieving high throughput on IP is hard and in part because it's never going to be super efficient at t…

You can use eBPF/DPDK these days for hardware offload.

Re: We need a replacement for TCP in the datacenter [pdf]

#219

Post by amluto »

cryptonector wrote: post #128
josephg wrote: post #125

> The data model for TCP is a stream of bytes. However, this is not the right data model for most datacenter applications. Datacenter applications typically exchange discrete messages to implement remote procedure calls This isn't just a datacenter problem. Every single network protocol I've ever created or implemented is message based, not stream based. Every messaging system. Every video game. Every RPC transport.…

> The only kinda-sorta exceptions I know about are HTTP/1.1 and telnet. But HTTP/1.1 is still a message oriented protocol; just with file-sized messages. (And even this stops being true with http2 anyway). No, HTTP/2 and QUIC do not change the semantics of HTTP. Also, you can have endless streams with HTTP/1.1: just use chunked encoding to POST/PUT and use Range: bytes=0- and chunked encoding for GET and chunked enco…

Once chunked encoding is in the picture, even HTTP/1.1 sends messages, not streams, under the hood.

Re: We need a replacement for TCP in the datacenter [pdf]

#220

Post by kijiki »

rossmohax wrote: post #193

> It uses several techniques for this, of which the most notable is that it takes advantage of the priority queues provided by modern switches. I'd understand if they said routers, but switches? Do L2 switches have any notion of priority and if yes how does it work?

Typically you map a VLAN to a priority via PFC (Priority Flow Control). You can do it with vconfig on Linux. Switch OSes have their own CLI for this.

Some switches can do PFC for untagged packets. They classify based on DSCP, and map that to a PFC priority. I've never used that though.

Post reply on HN