Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a replacement for TCP in the datacenter [pdf]

web.stanford.edu

Post reply on HN
Print view View original

131–140 of 329 posts

Re: We need a replacement for TCP in the datacenter [pdf]

#131

Post by lamontcg »

alexgartrell wrote: post #127
oconnor663 wrote: post #87

Earlier quoted context omitted.

> 50 years of clock time and thousands of years of engineer time It's not just the size of the investment, it's that it's the protocol everyone uses to talk to other people's machines, and you can't upgrade or replace other people's machines.

In this case we're talking about within the Datacenter, and you could conceivably update every network device and system to talk the new thing if you wanted. This is more achievable at a hyperscalar, where there tends to be TCP gives you three things: 1. Reasonable performance - This is hard but not impossible to replicate 2. Reliability - This is very hard to replicate because networking edge cases are very hard to…

After having lived through Amazon's early (pre-2003ish) UDP-based networking I got a laugh around 2006-ish or so reading about how facebook was into UDP. I assume there are people who worked there who still have the scars.

Re: We need a replacement for TCP in the datacenter [pdf]

#133

Post by topranks »

trustingtrust wrote: post #132

The original darpa paper is what the TCP/IP stack is still based on right ? It feels like it was never even intended to be at the scale its deployed on. Which amazes me to be honest that people have somehow gotten it to work to this scale.

The congestion control algorithms have been significantly improved since then. But yes it’s a testament to how good the original design is that it’s still what we use today.

Protocols like SCTP and QUIC work similarly but can avoid head-of-like blocking.

Re: We need a replacement for TCP in the datacenter [pdf]

#134

Post by klabb3 »

Areading314 wrote: post #64
fragmede wrote: post #45

Earlier quoted context omitted.

What is QUIC in your book? Given, say, $50 million of dev time, what would you go about fixing? And in what way?

Doesn't QUIC still run over TCP? I thought it was a replacement for HTTP not TCP (Edit: looks like it replaces TCP and HTTP)

It replaces TCP+TLS, and runs multiple streams on the same conn, supports transition from eg wifi to ethernet on at least one of the nodes. And since it's over udp implementations are mostly in user space. Which is good if you want it now, but not great for performance. Ip packets are very small so you gotta have either kernel support for quic or batch IO, otherwise it's often CPU limited (yes, really). In addition congestion control is wonky, unfortunately. In my experience (quic-go), it's too shy in the presence of TCP streams, which ends up getting more bandwidth. But that depends on the algorithm used, implementation and God knows what else.

Re: We need a replacement for TCP in the datacenter [pdf]

#135

Post by Luker88 »

A few years ago (when QUIC was coming out) I was developing the theory of a new transport/encryption/authentication protocol. The focus was as much on transport as in the built-in federated authentication.

There was not much interest in the field and I had a lot of the theory and formal proofs, but no implementation.

This month I found a cofounder and we are reordering a lot of the information and presentation, we should start asking for funds in more or less a month.

I still believe my solution to be much more complete than anything in use today (again: on paper), but since there seems to be some interest today, I'll ask here: Can anyone suggest some seed funds to check for a starttup? We will be based half US, half EU.

For more details, fenrirproject.org (again: old stuff there, ignore the broken code)

Re: We need a replacement for TCP in the datacenter [pdf]

#136

Post by topranks »

ghshephard wrote: post #129
tptacek wrote: post #15

"We hypothesize that flow-consistent routing is responsible for virtually all of the congestion that occurs in the core of datacenter networks". Flow-consistent routing is the constraint that packets for a given TCP 4-tuple get routed through the same network path, rather than balanced across all viable paths; locking a flow to a particular path makes it unlikely that segments will be received out of order on the des…

This is true, and the congestion mentioned here was subtle and not called out - typically flows are handled in a stateless manner by load balancers that hash on some set of MAC/IP/PORT features of the packet. This is where congestion occurs and the paper mentions it here: All that is needed for congestion is for two large flows to hash to the same intermediate link; this hot spot will persist for the life of the flow…

“Elephant” flows are a definitely a thing.

It all depends on the application and overall use in of the network.

With sufficient flows and a mix of sizes it’ll still tend to even out. But if you’ve significant high-throughout, long lived flows this is definitely something you might hit.

Re: We need a replacement for TCP in the datacenter [pdf]

#137

Post by ghshephard »

rkeene2 wrote: post #86

I wrote a blog post that might be interesting to those wanting an introduction into some of the basics of the problems called "02-FEB-2011: Why is there packet loss ?" [0] [0] https://rkeene.org/projects/info/wiki/176

Great post. I like the closing:

    Ultimately, we decided the best thing to do was to do nothing and hope for the best.
All of the experienced network engineers that I've worked with who have run into issues that feel like they can be improved by tweaking QoS somewhat, always end up saying to me, "Nah. Just get a bigger/more pipes." I've never been in a position like yours to make a cogent argument as to why they were wrong, and laid out the details as well as you did there.

Re: We need a replacement for TCP in the datacenter [pdf]

#139

Post by idlehand »

bsder wrote: post #109
throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. And yet every time hardware designers get the chance they redesign Ethernet and IPv4--poorly. See: HDMI 2.0+, USB 3.0+, Thunderbolt 3.0+, etc. My suspicion is that this paper works fine beween pairs of peers and immediately goes straight to hell after that. It is extremely suspicious that there is…

Never thought about that before. Ethernet supports extremely high levels of data transmission. ISB C for intrgrated charging ND data transfer makes sense, but why are there HDMI cables?

Re: We need a replacement for TCP in the datacenter [pdf]

#140

Post by jeroenhd »

intelVISA wrote: post #32
wmf wrote: post #28

Earlier quoted context omitted.

You can use SCTP in a datacenter. Ousterhout el al. are surely aware of SCTP so I assume Homa is better in some way.

Forgive my ignorance but why isn't SCTP more frequently used in DCs? I know it misbehaves with home routers etc. but shouldn't be a factor here.

Same reason Homa isn't used: software isn't written for it.

With SCTP there's also a significant performance impact because many drivers for the protocol are far from optimised, because very few applications use it, because of its performance implications, because very few programs use it, etc. etc.

There's also firewall issues: big firewall vendors just don't play nice with anything that's not a variant of HTTP(S). You still need some kind of firewall in a datacenter and it'd be foolish to set up two different ones for internal and external networking. Protocol ossification is real and if you use any external piece of firewall kit, you're sure to run into problems if you try to use "novel" protocols like SCTP. Hell, you'll be lucky to get good IPv6 support.

You can write your own access control if you want but that's often perceived as more expensive than buying a box, especially if the box companies find their way into a meeting with management.

Lastly, there's education. A shocking amount of developers have no idea about how networking works. They probably know there are protocols like UDP and TCP but their role and inner workings are often glazed over in my experience. Practical networking courses seem to treat the network as some kind of black box where bytes and IP addresses go in and response data comes out. If developers do know their basic networking, that information is often out of date; people don't seem to realise how often TCP gets tweaked to behave slightly differently to improve performance. Ask your average dev something about IPv6 and I doubt they'll know much more than "it's IPv4 with more bits" because networking simply doesn't come up that often.

In the end, it comes down to tradeoffs, experience, and decisions. Feel free to write SCTP code for your servers products where you can, the protocol definitely solves many issues people run into in TCP, but you'll probably have to defend your use of something unfamiliar to many developers every step along the way. The same is true for protocols like QUIC (outside the HTTP(S) environment) which tries to solve a whole lot of layer 3 to layer 5 problems in a single protocol that's designed to play nicely with shitty middleware boxes by its basis in UDP.

Post reply on HN