Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's time to replace TCP in the datacenter (2023)

arxiv.org

Post reply on HN
Print view View original

71–80 of 160 posts

Re: It's time to replace TCP in the datacenter (2023)

#71

Post by creshal »

wejick wrote: post #46
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

I'm imagining having a shared memory mounted as block storages then do the RPC thru this block. Some synchronization and polling/notifications work will need to be done.

The literal version of this is used by sanlock et al. to implement cluster-wide locks. But the whole "pretending to be block devices" overhead makes it not exactly ideal for anything else.

Drop the "pretending to be block devices" part and you basically end up with InfiniBand. It works well, if you ignore the small problem of "you need to either reimplement all your software to work with RDMA based IPC, or reimplement Ethernet on top of InfiniBand to remain compatible and throw away most advantages of InfiniBand again".

Re: It's time to replace TCP in the datacenter (2023)

#72

Post by bayindirh »

andrewflnr wrote: post #18
akira2501 wrote: post #8

> If Homa becomes widely deployed, I hypothesize that core congestion will cease to exist as a significant networking problem, as long as the core is not systemically overloaded. Yep. Sure; but, what happens when it becomes overloaded? > Homa manages congestion from the receiver, not the sender. [...] but the remaining scheduled packets may only be sent in response to grants from the receiver I hypothesize it will no…

Will it be a worse day than it would be with TCP? Either way, the only solution is to add more hardware, unless I'm misunderstanding the term "systemically overloaded".

I think so. If your core saturates, you add more capacity to your core switch. In HOMA, you need to add more receivers, but if you can't add them because the core can't handle more ports?

Ehrm. Looks like core saturation all over again.

Edit: Just popped to my mind. What prevents maliciously reducing "receive quotas" on compromised receivers to saturate an otherwise capable core? Looks like it's a very low bar for a very high impact DOS attack. Ouch.

Re: It's time to replace TCP in the datacenter (2023)

#73

Post by jbverschoor »

Sebb767 wrote: post #34
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

> I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter But it is often used for block storage in datacenters. Using it for anything else is going to be hard, as it is incompatible with TCP. The problem with not using TCP is the same thing HOMA will face - anything already speaks TCP, nearly all potential hires know TCP and most problems you have with TCP have been solved by smart engineers…

But you might not need TCP. For example, using file-sockets between an app, db, and http server (rails+pgsql+nginx for example) has many benefits. The beauty of OSI layers.

Re: It's time to replace TCP in the datacenter (2023)

#74

Post by KaiserPro »

TLDR: No, not its not.

HOMA is great, but not good enough to justify a wholesale ripout of TCP in the "datacentre"

Sure a lot of traffic is message oriented, but TCP is just a medium to transport those messages. Moreover its trivial to do external requests with TCP because its supported. There is not a need to have HOMA terminators at the edge of each datacentre to make sure that external RPC can be done.

The author assumes that the main bottleneck to performance is TCP in a datacentre. Thats just not the case, in my datacentre, the main bottleneck is that 100gigs point to point isnt enough.

Re: It's time to replace TCP in the datacenter (2023)

#75

Post by fmajid »

wejick wrote: post #46
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

I'm imagining having a shared memory mounted as block storages then do the RPC thru this block. Some synchronization and polling/notifications work will need to be done.

That’s essentially what RDMA is, except it is usually run over Infiniband although hyperscalers are wary of Nvidia’s control over the technology and looking for cheaper Ethernet-based alternatives.

https://blogs.nvidia.com/blog/what-is-rdma/

https://dl.acm.org/doi/abs/10.1145/3651890.3672233

Re: It's time to replace TCP in the datacenter (2023)

#76

Post by michaelt »

andrewflnr wrote: post #21
wmf wrote: post #6

Earlier quoted context omitted.

If they keep it secret they don't get credit for it.

How do you figure? The right decision is the right decision, even if you don't tell people. (granting, for the sake of argument, that it is the right decision)

When you are an outsider it's wise to take such claims with a grain of salt, because as the "secret" made its way to you, recounted by one person to another to another, there might have been an exaggeration, over-simplification or misunderstanding.

It's easy to imagine how, in the hands of tech journalists and youtubers optimising for clicks, "Google likes QUIC" and "Some ML clusters use infiniband" could get distorted into several faangs and the complete elimination of TCP.

Re: It's time to replace TCP in the datacenter (2023)

#77

Post by nine_k »

pif wrote: post #49

> Although Homa is not API-compatible with TCP, IPv6 anyone? People must start to understand that "Because this is the way it is" is a valid, actually extremely valid, answer to any question like "Why don't we just switch technology A with technology B?" Despite all the shortcomings of the old technology, and the advantages of the new one, inertia _is_ a factor, and you must accept that most users will simply even re…

I'd say that usually it's not about the balance of advantages but the balance of pain. You go through the pains of switching to a new and unfamiliar solution if your current solution is giving you even more pain.

If you don't feel much pain, you can and should stay with your current solution. If it's not broken, or not broken badly enough, don't fix it by radical surgery.

Re: It's time to replace TCP in the datacenter (2023)

#78

Post by holowoodman »

UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

Fibrechannel is far too expensive, you need expensive switches, cables/transceivers and cards in addition to the Ethernet you'll need anyways. And this Fibrechannel hardware is quite limited in what you can do with it, by far not as capable as the usual Ethernet/IP stuff with regards to routing, encryption, tunneling, filtering and what not.

Similar things are happening with stuff like Infiniband, it has become far too expensive and Ethernet/ROCE is making inroads in lower- to medium-end installations. Availability is also an issue, Nvidia is the only Infiniband vendor left.

Re: It's time to replace TCP in the datacenter (2023)

#79

Post by jpgvm »

parasubvert wrote: post #37

This has already been done at scale with HTTP/3 (QUIC), it's just not widely distributed beyond the largest sites & most popular web browsers. gRPC for example is still on multiplexed TCP via HTTP/2, which is "good enough" for many. Though it doesn't really replace TCP, it's just that the predominant requirements have changed (as Ousterhout points out). Bruce Davie has a series of articles on this: https://systemsapp…

Plus in a modern DC you can trivially convert it to be essentially lossless with DCSP/PFC and ECN which both work perfectly with any UDP based protocol (and is why NVMeOF, FCoE and RoCEv2 all woke so well today).

ECN isn't a necessity unless you need truly lossless network but the rest should get you pretty far as long as you are reasonably careful about communication patterns and blocking ratio of spine/core.

For anything that really needs the lowest possible latency at the cost of all other considerations there is still always Infiniband.

Re: It's time to replace TCP in the datacenter (2023)

#80

Post by bluGill »

holowoodman wrote: post #78
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

Fibrechannel is far too expensive, you need expensive switches, cables/transceivers and cards in addition to the Ethernet you'll need anyways. And this Fibrechannel hardware is quite limited in what you can do with it, by far not as capable as the usual Ethernet/IP stuff with regards to routing, encryption, tunneling, filtering and what not. Similar things are happening with stuff like Infiniband, it has become far t…

there is ip over fiber channel. no need for separate ethernet. At least in theory, in practice I'm sure if anyone implemented enough parts to make it useful but the spec exists.
Post reply on HN