Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a replacement for TCP in the datacenter [pdf]

web.stanford.edu

Post reply on HN
Print view View original

301–310 of 329 posts

Re: We need a replacement for TCP in the datacenter [pdf]

#301

Post by still_grokking »

KaiserPro wrote: post #241
pclmulqdq wrote: post #228

Earlier quoted context omitted.

The reality of today's large datacenters is that almost all of them have almost all of their traffic on TCP unless the owners of the datacenter have made a conscious effort to not use TCP. The highest-traffic applications, usually databases and storage systems, pretty much all use TCP unless you are buying a purpose-built HPC scale-out storage system (like a Lustre cluster). Most people who build a datacenter today u…

Fibre channel is dead, long live fibre channel. I agree a lot of things are on TCP, but I don't think its a massive problem, unless you are running close to the limit of your core network. And one solution to that is to upgrade your core network.... Failing that, implementing some load balancing/partitioning systems to make sure data-processing affinity is best matched. This the better solution, because it yields oth…

What is supposed to be a good scheduler?

Genuine question as I (as a software dev) have no clue how modern DCs are build in detail.

Re: We need a replacement for TCP in the datacenter [pdf]

#302

Post by Bluecobra »

kazen44 wrote: post #294
Bluecobra wrote: post #231

Earlier quoted context omitted.

> I’ve found UDP to be great for latency but pretty awful for throughout. UDP/multicast can provide excellent throughput. It's the de facto standard for market data on all major financial exchanges. For example, the OPRA feed (which is a consolidated market data feed of all options trading) can easily burst to ~17Gbps. Typically there is a "A" feed and a "B" feed for redundancy. Now you're talking about ~34Gbps of da…

Multicast throughput is hard to measure because it is... well, multicast. Depending on where your RP's are, and how you are transmitting multicast packets across a core, multicast performance can vary a lot. The main advantage of multicast however, is that throughput between RP's doesn't need to be very large..

It’s actually pretty easy to monitor the throughout with the right tools. The network capture appliance I use can measure microbursts at 1ms time intervals. With low latency/cut through switches there are limited buffers by design. You are certain to drop packets if you are trying to subscribe to a feed that can burst to 17Gbps on a 10Gbps port.

Market data typically comes from the same RP per exchange in most cases. Some exchanges split them by product type. Typically there’s one or two ingress points (two for redundancy) into your network at a cross connect in a data center.

Re: We need a replacement for TCP in the datacenter [pdf]

#303

Post by still_grokking »

wmf wrote: post #266
KaiserPro wrote: post #206

I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…

You're missing the fact that Stanford is the farm team for Google and Google is hyperscale. At scale, your "just spend more money" solutions are in fact more expensive than creating a new protocol. And like k8s, the new protocol can be sold to startups so they can "be like Google".

Google does not use K8s internally.

They never did, they won't ever do that!

K8s does not scale. Especially not to "Google scale".

First step to "be like Google" would be to ditch all that (docker-like) "container" madness and just compile static binaries. Than use something like Mesos to distribute workloads. Build literally everything as custom made on purpose solutions, and avoid mostly anything off the shelf.

"Being like Google" means not using any third party cloud stuff, but build your own in-house.

But this advice wouldn't sell GCP accounts. So Google does not tell you that. They telling you instead some marketing balderdash "how to be like Google".

Re: We need a replacement for TCP in the datacenter [pdf]

#304

Post by Bluecobra »

adamcharnock wrote: post #300
Bluecobra wrote: post #231

Earlier quoted context omitted.

> I’ve found UDP to be great for latency but pretty awful for throughout. UDP/multicast can provide excellent throughput. It's the de facto standard for market data on all major financial exchanges. For example, the OPRA feed (which is a consolidated market data feed of all options trading) can easily burst to ~17Gbps. Typically there is a "A" feed and a "B" feed for redundancy. Now you're talking about ~34Gbps of da…

That’s interesting. And I’m sure they have some very knowledgable people working for them who may(/will) know things I don’t. That being said, it wouldn’t surprise me if they were pushing 17G of UDP on 100G transports. Probably with some pretty high-end/expensive network hardware with huge buffers. I.e you can do it if you’ve got the money, but I bet TCP would still have better raw throughput.

Yep, 100G switches are common nowadays since the cost has come down so much, and you can easily carve a port to 4x10G, 4x25G, and 40G. In financial trading you tend to avoid switches with huge buffer as that comes to a huge cost in latency. For example, 2 megabytes of buffer is 1.68ms of latency on a 10G switch which is an eon in trading. Most opt for cut-through switches with shallow buffers measured in 100s of nanoseconds. If you want to get really crazy there are L1 switches that can do 5ns.

Re: We need a replacement for TCP in the datacenter [pdf]

#305

Post by benlivengood »

KaiserPro wrote: post #274
wmf wrote: post #266

Earlier quoted context omitted.

You're missing the fact that Stanford is the farm team for Google and Google is hyperscale. At scale, your "just spend more money" solutions are in fact more expensive than creating a new protocol. And like k8s, the new protocol can be sold to startups so they can "be like Google".

You're missing the point that maybe, just maybe, I'm part of a team that looks after >5 million servers. You might also divine that while TCP can be a problem, a bigger problem is data affinity. Shuttling data from a next door rack costs less than one that's in the next door hall, and significantly less than the datacentre over. With each internal hop, the risk of congestion increases. You might also divine that chan…

> quick maths: a faster top of rack switch is possibly the same cost as 5 days engineering wage for a mid level google employee. How many new switches do you think you could buy with the engineering effort required to port everything to the new protocol, and have it stable and observable?

So your 5M machines / 40 in the best case of all 1U boxes is 125K TOR-switch-SWE-week-equivalents / 52 weeks in a year which comes to 2K SWE-years to invest in new protocols, observability, and testing. Google got to the scale they are by explicitly spending on SWE-hours instead of Cisco.

Re: We need a replacement for TCP in the datacenter [pdf]

#306

Post by still_grokking »

throwaway892238 wrote: post #37

Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…

> There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo.

I would affirm that. It's imho true for almost everything in IT tech.

How computers "work" today is just pure madness when looked anyhow closer.

Everything's a result of some "historic accidents" back in the days, and from that the usual race to the bottom caused by market powers.

Nobody is willing to touch any of the lower layers no matter how crazy they are form today's viewpoint. We just shovel new layers on top to paper over the mistakes of the past. Nothing gets repaired, or actually what would be more more important, rethought form the ground up in light of new technological possibilities and changed requirements.

I understand from the economic standpoint how this comes. But I'm also quite sure we didn't make any fundamental improvements in the last 50 years of computing.

That's a very bad sign when everything in a field that's not even really 100 years old is frozen in time since 50 years because everything's so fragile and complex that fundamental changes aren't possible. This looks like a text book example of a house of cards…

Given how vital IT tech is to modern life I fear that this will crash at some point in the worst way possible.

And even if it won't crash, which is really strongly hope, we will never have nice things again as nothing of the old rotten things can be reasonably changed.

Re: We need a replacement for TCP in the datacenter [pdf]

#307

Post by lamontcg »

Bluecobra wrote: post #233
nextstepguy wrote: post #204

Earlier quoted context omitted.

Multicast storms happened regularly back in 2004

True, there were tons of crappy hardware still in production at that time. The first job I had out of college consisted of crappy 3Com hubs (not switches) so something like Norton Ghost could take down the whole network since multicast would get flooded everywhere. Nowadays this is a less of a problem as hubs are long gone and most switches have IGMP snooping by default and would only forward mutlicast frames that so…

Amazon was definitely not run off of crappy 3com hubs, not even back then.

Re: We need a replacement for TCP in the datacenter [pdf]

#308

Post by MuffinFlavored »

wmf wrote: post #277
MuffinFlavored wrote: post #89

Earlier quoted context omitted.

Dumb question, there's no way to talk to a PC over the Internet with Homa, right? Since our home ISPs + routers are all only doing UDP/TCP over IPv4/IPv6? Homa is mainly for "LAN"?

Yes.

Will we ever see the day where computers over the Internet talk to each other over something other than UDP/TCP over IPv4/IPv6?

Re: We need a replacement for TCP in the datacenter [pdf]

#309

Post by jimmySixDOF »

signa11 wrote: post #62
jimmySixDOF wrote: post #31

Earlier quoted context omitted.

Yes that stat stood out for me too and I was wondering how to actually test this without breaking anything in the process.

> Yes that stat stood out for me too and I was wondering how to actually test this without breaking anything in the process. dpdk has been doing just that for quite some time now. perhaps you can try that and see ?

Thanks I will look into it but on first glance their testing still seems like its under lab conditions. Thanks for the tip!

Re: We need a replacement for TCP in the datacenter [pdf]

#310

Post by musicale »

Haven't data centers switched to RDMA already? Why are we still wasting time with this networking nonsense when most of the time we're just copying data from one memory or cache to another, over a private interconnect? ;-)

It seems that Osterhout's complaints about RDMA are about current RDMA implementations. I expect many of them are fixable.

Osterhout's complaints about TCP are all valid, though he doesn't mention my pet peeve with TCP, which is that connectivity breaks when your IP address changes. And requiring apps to deal with IP addresses in user-level APIs seems like a mistake.

Simple request-response RPC protocols were a good idea in the 1980s, and they're still a good idea today. I should probably read the Homa paper(s) regarding congestion control though, as it isn't covered in this PDF.

Post reply on HN