Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's time to replace TCP in the datacenter (2023)

arxiv.org

Post reply on HN
Print view View original

141–150 of 160 posts

Re: It's time to replace TCP in the datacenter (2023)

#141

Post by UltraSane »

Sylamore wrote: post #128
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

InfiniBand would make more sense than Fibre Channel

Maybe but InfiniBand is really expensive and InfiniBand switches are in short supply. And it is an L2 protocol while FC is L3

Re: It's time to replace TCP in the datacenter (2023)

#142

Post by UltraSane »

KaiserPro wrote: post #69
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

FC was much more expensive than ethernet, so needed a reason to be used. For block storage it is great, if slower than ethernet.

Is their a fundamental reason for it being more expensive than Ethernet or is just greed?

Re: It's time to replace TCP in the datacenter (2023)

#143

Post by UltraSane »

markhahn wrote: post #103
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

the question is really: does it have anything vendor-specific, interop-breakers? FC seems to work nicely in a single-vendor stack, or at least among specific sets of big-name vendors. that's OK for the "enterprise" market, where prices are expected to be high, and where some integrator is getting a handsome profit for making sure the vendors match. besides consumer, the original non-enterprise market was HPC, and we…

FC speeds have really lagged. 64Gbps is available but not widely used and 128Gbps was introduced in 2023. But since by definition 100% of FC bandwidth can only be used for storage it has been adequate.

https://fibrechannel.org/preview-the-new-fibre-channel-speed...

Re: It's time to replace TCP in the datacenter (2023)

#144

Post by UltraSane »

holowoodman wrote: post #78
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

Fibrechannel is far too expensive, you need expensive switches, cables/transceivers and cards in addition to the Ethernet you'll need anyways. And this Fibrechannel hardware is quite limited in what you can do with it, by far not as capable as the usual Ethernet/IP stuff with regards to routing, encryption, tunneling, filtering and what not. Similar things are happening with stuff like Infiniband, it has become far t…

Is there a fundamental reason why FC is more expensive than Ethernet?

Re: It's time to replace TCP in the datacenter (2023)

#145

Post by convolvatron »

UltraSane wrote: post #144
holowoodman wrote: post #78

Earlier quoted context omitted.

Fibrechannel is far too expensive, you need expensive switches, cables/transceivers and cards in addition to the Ethernet you'll need anyways. And this Fibrechannel hardware is quite limited in what you can do with it, by far not as capable as the usual Ethernet/IP stuff with regards to routing, encryption, tunneling, filtering and what not. Similar things are happening with stuff like Infiniband, it has become far t…

Is there a fundamental reason why FC is more expensive than Ethernet?

since its entire reason to exist was to effect an artificial market segmentation, I guess the answer is .. yes?

Re: It's time to replace TCP in the datacenter (2023)

#146

Post by aseipp »

parasubvert wrote: post #37

This has already been done at scale with HTTP/3 (QUIC), it's just not widely distributed beyond the largest sites & most popular web browsers. gRPC for example is still on multiplexed TCP via HTTP/2, which is "good enough" for many. Though it doesn't really replace TCP, it's just that the predominant requirements have changed (as Ousterhout points out). Bruce Davie has a series of articles on this: https://systemsapp…

QUIC and Homa are not remotely similar and have completely different design constraints. I have no idea why people keep bringing up QUIC in this thread other than "It's another thing that isn't TCP." Yes, many things are not-TCP. The details are what matter.

Re: It's time to replace TCP in the datacenter (2023)

#147

Post by Shawnj2 »

stego-tech wrote: post #89

As others have already hit upon, the problem forever lies in standardization of whatever is intended to replace TCP in the data center, or the lack thereof. You’re basically looking for a protocol supported in hardware from endpoint to endpoint, including in firewalls, switches, routers, load balancers, traffic shapers, proxies, etcetera - a very tall order indeed. Then, to add to that very expensive list of criteria…

Obvious answer is to run something on top of UDP like QUIC does

Re: It's time to replace TCP in the datacenter (2023)

#148

Post by YZF »

UltraSane wrote: post #56
YZF wrote: post #39

Earlier quoted context omitted.

Are you suggesting some protocol layer of Fibre Channel to be used over IP over Ethernet? TCP (in practice) runs on top of (mostly) routed IP networks and network architectures. E.g. a spine/leaf network with BGP. Fibre Channel as I understand it is mostly used in more or less point to point connections? I do see some mention of "Switched Fabric" but is that very common?

Fibre Channel is a routed L3 protocol that can support loop-free multi-path typologies.

I'll admit I'm not familiar with the routing protocols used for Fibre Channel. Is there some equivalent of BGP? How well does it scale? What vendors sell FC switches and what's the cost compared to Ethernet/IP/TCP?

Re: It's time to replace TCP in the datacenter (2023)

#149

Post by UltraSane »

convolvatron wrote: post #145
UltraSane wrote: post #144

Earlier quoted context omitted.

Is there a fundamental reason why FC is more expensive than Ethernet?

since its entire reason to exist was to effect an artificial market segmentation, I guess the answer is .. yes?

It isn't an artificial market segmentation. Fibre Channel is a no compromise technology with a single purpose, to connect servers to remote storage with performance and reliability close to directly attached storage, and it does that really, REALLY well. It is by far the single most bullet proof technology I have ever used. In a parallel universe where FC won over ethernet and every Ethernet port in the world was an FC port I don't see why it would be any more expensive that ethernet.

Re: It's time to replace TCP in the datacenter (2023)

#150

Post by UltraSane »

YZF wrote: post #148
UltraSane wrote: post #56

Earlier quoted context omitted.

Fibre Channel is a routed L3 protocol that can support loop-free multi-path typologies.

I'll admit I'm not familiar with the routing protocols used for Fibre Channel. Is there some equivalent of BGP? How well does it scale? What vendors sell FC switches and what's the cost compared to Ethernet/IP/TCP?

FC uses Fabric Shortest Path First, which is a lot like OSPF. It can scale to 2^64 ports. There is no FC equivalent of BGP. Broadcom, Cisco, HP, Lenovo, IBM sell FC switches, but some of them are probably rebadged Broadcom switches. The worst thing about FC is that the switches are licensed per port so you might not be able to use all the ports on the device. A brocade G720 with 24 32Gb usable ports is $28,000 on CDW. It has 64 physical ports. a 24 port license is $31,000 on CDW. So it is REALLY freaking expensive. But for servers a company can't make money without it is absolutely worth it. One place I worked had a old EMC FC SAN with 8 years of 100% uptime.
Post reply on HN