Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a replacement for TCP in the datacenter [pdf]

web.stanford.edu

Post reply on HN
Print view View original

201–210 of 329 posts

Re: We need a replacement for TCP in the datacenter [pdf]

#201

Post by iso1631 »

simplotek wrote: post #146
duped wrote: post #112

Earlier quoted context omitted.

Every major corporation has multiple research organizations doing nothing but invest in things that don't have immediate shareholder value. What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale. The scale at which these changes needs to happen to be practical are at the cluster level in a single data center. If you ca…

> What you're talking about though isn't just coming up with new ideas or even new products. It's replacing hundreds of billions in infrastructure wholesale. I'd put it differently: it's paying up hundreds of billions in infrastructure to have some sort of gain. And which gain is that exactly? I see a lot of "the world is dumb but I am smart" comments in this thread but I saw no one presenting any clear advantage or…

> Every single criticism of TCP/IP in this thread sounds like change for the sake of change, and changes that aren't even presented with tangible improvements in mind or a clear performance gain.

It amuses me that many of those saying "we need to change" are the same ones that bemoan it when car manufacturers remove buttons or make glove boxes operational from a touch screen because they can.

Re: We need a replacement for TCP in the datacenter [pdf]

#203

Post by askvictor »

Kiro wrote: post #200
8371861215 wrote: post #188

Earlier quoted context omitted.

Quoted post unavailable.

Very peculiar spam. Does anyone have a theory what the motive is?

Modern equivalent of a Numbers Station https://en.wikipedia.org/wiki/Numbers_station

Re: We need a replacement for TCP in the datacenter [pdf]

#204

Post by nextstepguy »

lamontcg wrote: post #131
alexgartrell wrote: post #127

Earlier quoted context omitted.

In this case we're talking about within the Datacenter, and you could conceivably update every network device and system to talk the new thing if you wanted. This is more achievable at a hyperscalar, where there tends to be TCP gives you three things: 1. Reasonable performance - This is hard but not impossible to replicate 2. Reliability - This is very hard to replicate because networking edge cases are very hard to…

After having lived through Amazon's early (pre-2003ish) UDP-based networking I got a laugh around 2006-ish or so reading about how facebook was into UDP. I assume there are people who worked there who still have the scars.

Multicast storms happened regularly back in 2004

Re: We need a replacement for TCP in the datacenter [pdf]

#205

Post by Tor3 »

adamcharnock wrote: post #168
tezza wrote: post #155

Earlier quoted context omitted.

Do you have any specific problems you can elaborate with the UDP ? UDP used successfully many places.

I’m really looking forwarding to seeing the original commenters reply on this. But I’ll share my experience too. I’ve found UDP to be great for latency but pretty awful for throughout. Especially over longer routes (ie inter-region transports). Also, if you fire UDP packets out of a machine in a tight loop then there is every chance you could overload various buffers and just loose them (depending on the networking h…

Anecdotal, but I've some experience in running both TCP- and UDP-based VPN over long-latency links (I worked from half around the globe for some years).

With OpenVPN it's easy enough to test - configure for UDP, or configure for TCP. With long latency, and a tiny amount of packet losses, running TCP over TCP OpenVPN completely stalls, while TCP over UDP OpenVPN is excellent - it's around the same performance as running direct TCP, or sometimes actually better. At work we've also used other types of VPN setups (for engineers on the road), and the TCP based ones (we've used several) work fine most of the time, but if you try that from far away it becomes nearly unusable while UDP OpenVPN continues to work basically just fine.

The TCP over TCP VPN performance problem (over long latency links) presumably has to do with windowing and ack/nak on top of windowing with ack/nak.

Re: We need a replacement for TCP in the datacenter [pdf]

#206

Post by KaiserPro »

I get where this is coming from, but no. We don't need to replace TCP in the datacentre.

Why?

because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP)

Look if it really was that much of a problem then everyone in datacentres would move to RDMA over infiniband. For shared memory clusters, thats what's been done for years. but for general purpose computing its pretty rare. Most of the time its not worth the effort. Infiniband is cheap now, so its not that hard to deploy RDMA[1] type interconnects. Having a reliable layer2 with inbuilt flow control solves a number of issues, even if you are just slamming IP over the top.

shit, even 25/100gig is cheap now. so most of your problems can be solved by putting extra nics in your servers and have fancypants distributed LACP type setups on your top of rack/core network.

The biggest issue is that its not the network that's constraining throughput, it either processing or some other non network IO.

[1]I mean it is hard, but not as hard as implementing a brand new protocol and expecting it to be usable and debuggable.

Re: We need a replacement for TCP in the datacenter [pdf]

#207

Post by dooglius »

bsder wrote: post #109
throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. And yet every time hardware designers get the chance they redesign Ethernet and IPv4--poorly. See: HDMI 2.0+, USB 3.0+, Thunderbolt 3.0+, etc. My suspicion is that this paper works fine beween pairs of peers and immediately goes straight to hell after that. It is extremely suspicious that there is…

Why do you say that these protocols are worse than Ethernet/IPv4? I'm not intimately familiar with any at L2/L3, but I don't think any have hacks as bad as ARP. (USB does have some weirdness at L1 though I know.)

Re: We need a replacement for TCP in the datacenter [pdf]

#208

Post by nickdothutton »

23 years ago I sat in a meeting with Sun, Intel, Mellanox, and 1 or 2 others. In that meeting we discussed putting an RDMA interface on individual hard drives, trays of RAM, CPUs, and other more exotic devices (like battery backed RAM, no conventional SSD in those days of course). You’d install RAM 1 42U rack at a time, disks likewise, CPUs in another rack and so on. All partitioned, controlled, managed, and of course billed-for by a “data center OS”.

Re: We need a replacement for TCP in the datacenter [pdf]

#209

Post by andersced »

We have been testing out various protocols to overcome in our case TCP head-of-line blocking by using the protocols->

SRT: https://github.com/Haivision/srt (C++ wrapper https://github.com/andersc/cppSRTWrapper)

RIST: https://code.videolan.org/rist/rist-cpp

KCP: https://github.com/Unit-X/kcp-cpp

We wrap all data in a common container format https://github.com/agilecontent/efp

To decouple the data from the transport.

Yes the above solutions are media centric but can be used for almost any arbitrary data.

The protocols are not 'fare' so starvation may happen, and must be handled on the application level.

/A

Re: We need a replacement for TCP in the datacenter [pdf]

#210

Post by throwaway41597 »

kanwisher wrote: post #145
Luker88 wrote: post #135

A few years ago (when QUIC was coming out) I was developing the theory of a new transport/encryption/authentication protocol. The focus was as much on transport as in the built-in federated authentication. There was not much interest in the field and I had a lot of the theory and formal proofs, but no implementation. This month I found a cofounder and we are reordering a lot of the information and presentation, we sh…

Why don’t you actually build gasp a prototype before asking for money

Depending on the scope, it's not always possible to self-fund while starting a project.
Post reply on HN