Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need a replacement for TCP in the datacenter [pdf]

web.stanford.edu

Post reply on HN
Print view View original

321–329 of 329 posts

Re: We need a replacement for TCP in the datacenter [pdf]

#321

Post by tiberious726 »

adamcharnock wrote: post #317
Bluecobra wrote: post #304

Earlier quoted context omitted.

Yep, 100G switches are common nowadays since the cost has come down so much, and you can easily carve a port to 4x10G, 4x25G, and 40G. In financial trading you tend to avoid switches with huge buffer as that comes to a huge cost in latency. For example, 2 megabytes of buffer is 1.68ms of latency on a 10G switch which is an eon in trading. Most opt for cut-through switches with shallow buffers measured in 100s of nano…

That is a really good point that I hadn’t considered. Presumably this comes at the risk of dropped packets if the upstream link becomes saturated? Does one just size the links accordingly to avoid that?

Basically yes, but the links themselves are controlled by the exchanges (and tied in to your general contact for market access).

In general UDP is not a problem in the space because of overprovisioning. Think "algorithms are for people who don't know how to buy more RAM", but with a finicial industry budget behind it.

Re: We need a replacement for TCP in the datacenter [pdf]

#322

Post by tiberious726 »

Bluecobra wrote: post #304
adamcharnock wrote: post #300

Earlier quoted context omitted.

That’s interesting. And I’m sure they have some very knowledgable people working for them who may(/will) know things I don’t. That being said, it wouldn’t surprise me if they were pushing 17G of UDP on 100G transports. Probably with some pretty high-end/expensive network hardware with huge buffers. I.e you can do it if you’ve got the money, but I bet TCP would still have better raw throughput.

Yep, 100G switches are common nowadays since the cost has come down so much, and you can easily carve a port to 4x10G, 4x25G, and 40G. In financial trading you tend to avoid switches with huge buffer as that comes to a huge cost in latency. For example, 2 megabytes of buffer is 1.68ms of latency on a 10G switch which is an eon in trading. Most opt for cut-through switches with shallow buffers measured in 100s of nano…

Do the vendors actually convince anyone to buy those hubs rebranded as "ultra high performance 5ns L1 switches"?

Re: We need a replacement for TCP in the datacenter [pdf]

#323

Post by tiberious726 »

Bluecobra wrote: post #302
kazen44 wrote: post #294

Earlier quoted context omitted.

Multicast throughput is hard to measure because it is... well, multicast. Depending on where your RP's are, and how you are transmitting multicast packets across a core, multicast performance can vary a lot. The main advantage of multicast however, is that throughput between RP's doesn't need to be very large..

It’s actually pretty easy to monitor the throughout with the right tools. The network capture appliance I use can measure microbursts at 1ms time intervals. With low latency/cut through switches there are limited buffers by design. You are certain to drop packets if you are trying to subscribe to a feed that can burst to 17Gbps on a 10Gbps port. Market data typically comes from the same RP per exchange in most cases.…

Have you tried to get inline-timestamping going on those fancy modern NICs that support PPT? Orders of magnitude cheaper than new ingress ports on that appliance whose name starts with a "C", also _really_ cool to have more perspectives on the network than switch monitor sessions.

Re: We need a replacement for TCP in the datacenter [pdf]

#324

Post by KaiserPro »

still_grokking wrote: post #316
KaiserPro wrote: post #314

Earlier quoted context omitted.

Its very much down to your workload and how you want it to work. Short answer: k8s/fargate/ECS/batch will do what most people want. Personally I'd steer clear of k8s until you 100% need that overhead. Managed services are ok. Long answer: K8s has a whole bunch of scheduling algorithms but its a jack of all trades, and only really deals with very shallow dependencies (there are plugins but I've not used them). For Lik…

Thanks for the reply! Not really an answer to the scheduler question, but at least it mirrors some of my experience. That K8s is something to avoid, and that it does not scale, is a known (at least to me). But that doesn't answer what people would put on the metal when building DCs… I was not asking out of the perspective of an end-user. I was asking about (large) DC scale infra. (As dev I know the end-user stuff). A…

Sorry yeah, I didn't really answer your question.

Here is a non-exhaustive list of schedulers for differing use cases:

https://slurm.schedmd.com/documentation.html https://www.altair.com/grid-engine https://rmanwiki.pixar.com/display/TRA/Tractor+2 https://www.opencue.io/ https://abelay.github.io/6828seminar/papers/hazelwood:ml.pdf I'm assuming you've read about borg

As you've pointed out mesos is there as well.

Re: We need a replacement for TCP in the datacenter [pdf]

#325

Post by KaiserPro »

KaiserPro wrote: post #315
benlivengood wrote: post #305

Earlier quoted context omitted.

> quick maths: a faster top of rack switch is possibly the same cost as 5 days engineering wage for a mid level google employee. How many new switches do you think you could buy with the engineering effort required to port everything to the new protocol, and have it stable and observable? So your 5M machines / 40 in the best case of all 1U boxes is 125K TOR-switch-SWE-week-equivalents / 52 weeks in a year which comes…

> explicitly spending on SWE-hours instead of Cisco. I strongly doubt that TOR switches are cisco

but to answer your further case. The point is you don't need to replace all the TOR switches. Only the ones that deal with high network IO.

to change protocol you need gateways/loadbalancers either at the edge of the DC just after the public end points, or in the "high speed" areas that are running high network IO. For that to work, you'll need to show its worth the engineering effort/maintenance/latency.

Re: We need a replacement for TCP in the datacenter [pdf]

#326

Post by still_grokking »

KaiserPro wrote: post #324
still_grokking wrote: post #316

Earlier quoted context omitted.

Thanks for the reply! Not really an answer to the scheduler question, but at least it mirrors some of my experience. That K8s is something to avoid, and that it does not scale, is a known (at least to me). But that doesn't answer what people would put on the metal when building DCs… I was not asking out of the perspective of an end-user. I was asking about (large) DC scale infra. (As dev I know the end-user stuff). A…

Sorry yeah, I didn't really answer your question. Here is a non-exhaustive list of schedulers for differing use cases: https://slurm.schedmd.com/documentation.html https://www.altair.com/grid-engine https://rmanwiki.pixar.com/display/TRA/Tractor+2 https://www.opencue.io/ https://abelay.github.io/6828seminar/papers/hazelwood:ml.pdf I'm assuming you've read about borg As you've pointed out mesos is there as well.

Cool! Thanks! That's a lot of stuff I didn't hear about until now.

It's really nice that one can meet experts here on HN and get free valuable answers form them. Thank you.

Re: We need a replacement for TCP in the datacenter [pdf]

#327

Post by specialist »

KaiserPro wrote: post #274
wmf wrote: post #266

Earlier quoted context omitted.

You're missing the fact that Stanford is the farm team for Google and Google is hyperscale. At scale, your "just spend more money" solutions are in fact more expensive than creating a new protocol. And like k8s, the new protocol can be sold to startups so they can "be like Google".

You're missing the point that maybe, just maybe, I'm part of a team that looks after >5 million servers. You might also divine that while TCP can be a problem, a bigger problem is data affinity. Shuttling data from a next door rack costs less than one that's in the next door hall, and significantly less than the datacentre over. With each internal hop, the risk of congestion increases. You might also divine that chan…

> a bigger problem is data affinity

For future, please write about how typical cloud customers can design for better data affinity.

Or is it just handled by the provider?

FWIW, at a prev gig, knowing nothing about nothing, I finally persuaded our team to colocate a Redis process on each of our EC2 instances (along side the http servers). Quick & dirty solution to meet our PHBs silly P99 requirements (for a bog standard ecommerce site).

Apologies for belated, noob question.

Re: We need a replacement for TCP in the datacenter [pdf]

#328

Post by jl2718 »

cryptonector wrote: post #251
zokier wrote: post #198

Earlier quoted context omitted.

> This is a recipe for DDoS. Inside a datacenter?

Oh, right, in a datacenter probably not. EDIT: But you know, UDP meets your bill.

Datagram is a layer-3 protocol. There’s a lot going on underneath that.

Re: We need a replacement for TCP in the datacenter [pdf]

#329

Post by nsteel »

lamontcg wrote: post #290
tezza wrote: post #155

Earlier quoted context omitted.

Do you have any specific problems you can elaborate with the UDP ? UDP used successfully many places.

Amazon used UDP over multicast for request/response when sometimes the responses would be very large and implemented reliability on top of that through fall back to UDP unicast. This was all using Tibco RVD (taken from Bezos experience in Finance on the East Coast before Amazon I think). The really key point there is probably the size of the responses, it wasn't just tiny atomic bits of stock information. At one poin…

> Ethernet has no flow control

Isn't this what pause frames and Pfc are for? (Honest question)

Post reply on HN