Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's time to replace TCP in the datacenter (2023)

arxiv.org

Post reply on HN
Print view View original

131–140 of 160 posts

Re: It's time to replace TCP in the datacenter (2023)

#131

Post by andrewflnr »

michaelt wrote: post #124
andrewflnr wrote: post #92

Earlier quoted context omitted.

True as far as it goes, but "they didn't actually do it" is a different story from "they did it secretly". The two claims exclude each other, so you can't really compare them in the context of "credit".

Allow me to rephrase. I think the statement is not true, in a literal sense. I do not think there are multiple FAANG companies with data centres where TCP has been entirely eliminated. But the statement is ambiguous enough you could come up with true interpretations, if you diluted it to the point of meaninglessness. Therefore, without a clear public statement of what is being claimed, it's very difficult for me to b…

Very reasonable. It does seem dubious. In that case I just think my comment, itself already responding to a very different concern, was a weird place to raise that question. :)

Re: It's time to replace TCP in the datacenter (2023)

#132

Post by andrewflnr »

bayindirh wrote: post #104
andrewflnr wrote: post #95

Earlier quoted context omitted.

The "receivers" are just other machines in your data center. Their number is determined by the workload, same as always. Adding more will tend to increase traffic. I'm not a datacenter expert, but is "not enough ports" really a showstopper? It just sounds like bad planning to me. And still not a protocol issue.

Depends on where you run out of ports, actually. A datacenter has layers of networking, and some of it is what we call "the core" or "the spine" of the network. Sometimes you need to shuffle things around, and you need to move things closer to core, and you can't because there are no ports. Or you procure the machines, and adding new machines requires some ports closer or at the core, and you are already running at f…

Ok, that's good to know. But I still don't see how having congestion control be driven by receivers instead of senders makes it harder to fix than it is currently.

I mean, I don't actually see why you would need more ports at all. You still just have a certain number of machines that want to exchange a certain amount of traffic. That number is either above or below what your core can handle (guessing, again, at what the author means by "systemically overloaded").

Re: It's time to replace TCP in the datacenter (2023)

#134

Post by _zoltan_ »

tonetegeatinst wrote: post #108
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

Fiber is attractive. As someone who wants to upgrade to fiber, the main barrier to entry is the cost of switches and a router. Granted I'm also trying to find a switch that supports ROCm and rdma. Not easy to find a high bandwidth switch that supports this stuff without breaking the bank.

SN2100/2700 from eBay?

Re: It's time to replace TCP in the datacenter (2023)

#135

Post by stego-tech »

panzagl wrote: post #120
stego-tech wrote: post #99

Earlier quoted context omitted.

That’s my point: TCP in the datacenter remains a 1% problem, in the sense that only 1% of customers actually have this as a problem, and only 1% of those have the ability to invest in a solution. At that point, market conditions incentivize protecting their work and selling it to others (e.g., Public Cloud Service Providers) as opposed to releasing it into the wild as its own product line for general purchase (e.g.,…

Amazon's solutions to 1% problems are the next batch of cargo cult corporate solution speak that we're going to have to deal with. How can we have a web scale agile big data devops monorepo machine learning microservice oriented architecture if we're limited by TCP? I mean, it was developed in the 70s...

Ugh, the fact I have to explain that we don’t need any of that cult-speak for our enterprise VMs (because enterprise software hasn’t even moved to containers yet) makes me grimace. I loathe how marketers and salesfolk have turned technology from a thing that can be adapted to solve your problems, into a productized solution for problems you never knew you had until the magazine they advertise in told your CIO about it.

Re: It's time to replace TCP in the datacenter (2023)

#136

Post by stego-tech »

wbl wrote: post #112
stego-tech wrote: post #99

Earlier quoted context omitted.

That’s my point: TCP in the datacenter remains a 1% problem, in the sense that only 1% of customers actually have this as a problem, and only 1% of those have the ability to invest in a solution. At that point, market conditions incentivize protecting their work and selling it to others (e.g., Public Cloud Service Providers) as opposed to releasing it into the wild as its own product line for general purchase (e.g.,…

Nevertheless Infiband exists.

As does Fibre Channel and a myriad of other solutions out there. The point wasn’t to bring every “but X exists” or “but Company A said in this blog post they solved it” response out of the fog, but to point out that these issues are incredibly fringe to begin with yet make the rounds a few times a year every time someone has an “epiphany” about the inefficiencies of TCP/IP in their edgiest of edge-case scenarios.

TCP isn’t the most efficient protocol, sure, but it survives and thrives because of its flexibility, cost, and broad adoption. For everything else, there’s undoubtedly something that already exists to solve your specific gripe about it.

Re: It's time to replace TCP in the datacenter (2023)

#137

Post by stego-tech »

skeptrune wrote: post #123
stego-tech wrote: post #89

As others have already hit upon, the problem forever lies in standardization of whatever is intended to replace TCP in the data center, or the lack thereof. You’re basically looking for a protocol supported in hardware from endpoint to endpoint, including in firewalls, switches, routers, load balancers, traffic shapers, proxies, etcetera - a very tall order indeed. Then, to add to that very expensive list of criteria…

> Even if they do (e.g., Google with QUIC), the broad vibe I get is that folks aren’t likely to trust those offerings as lacking in ulterior motives. It's pretty unfortunate that we've landed here. Hordes of venture-backed companies building shareware-like software with an "open source" label has done some severe damage.

Which is ironic, because I remember TCP/IP maturing in the protocol wars of the 90s. My Cisco course specifically covered the protocols separately from the media layers because you couldn’t know if your future employer still leveraged Token Ring, or ATM, or IPX, or TCP; a decade later, the course had drastically simplified to “ethernet” and “TCP/IP” only.

Many of these came from companies who created the protocol solely to push products, which meant the protocols themselves had to compete outside of the vacuum chamber of software alone and instead operate in real world scenarios and product lines. This also meant that as we engineers and SysAdmins deployed them in our enterprises, we quite literally voted with our wallets where able on the gear and protocols that met our needs. Unsurprisingly, TCP/IP won out for general use because of its low cost of deployment and ongoing support compared to alternatives, and that point is lost on the modern engineer that’s just looking at this stuff as “paper problems”.

Re: It's time to replace TCP in the datacenter (2023)

#138

Post by shawn_w »

wutwutwat wrote: post #119
shawn_w wrote: post #116

Earlier quoted context omitted.

Unix domain stream sockets do not use tcp. Nor do unix datagram sockets use udp. They're much simpler.

>The type parameter should be one of two common socket types: stream or datagram.[10] A third socket type is available for experimental design: raw. > SOCK_STREAM will create a stream socket. A stream socket provides a reliable, bidirectional, and connection-oriented communication channel between two processes. Data are carried using the Transmission Control Protocol (TCP). > SOCK_DGRAM will create a datagram socket.…

Not the first time a Wikipedia article has been wrong. That one seems to be talking about IP sockets and local ones at the same time instead of focusing on local ones. Could definitely stand to be rewritten.

Re: It's time to replace TCP in the datacenter (2023)

#139

Post by UltraSane »

bluGill wrote: post #80
holowoodman wrote: post #78

Earlier quoted context omitted.

Fibrechannel is far too expensive, you need expensive switches, cables/transceivers and cards in addition to the Ethernet you'll need anyways. And this Fibrechannel hardware is quite limited in what you can do with it, by far not as capable as the usual Ethernet/IP stuff with regards to routing, encryption, tunneling, filtering and what not. Similar things are happening with stuff like Infiniband, it has become far t…

there is ip over fiber channel. no need for separate ethernet. At least in theory, in practice I'm sure if anyone implemented enough parts to make it useful but the spec exists.

No. When I heard about Cisco having FC over Ethernet for their UCS servers I was grossed out because of how Ethernet is a L2 protocol that can't handle multi-path without ugly hacks like Virtual Port Channel and discovered that there is no real support for IP over Fiber Channel. There is a wikipedia page for IPFC but it seems to be completely dead.

https://en.wikipedia.org/wiki/IPFC

Re: It's time to replace TCP in the datacenter (2023)

#140

Post by UltraSane »

tonetegeatinst wrote: post #108
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

Fiber is attractive. As someone who wants to upgrade to fiber, the main barrier to entry is the cost of switches and a router. Granted I'm also trying to find a switch that supports ROCm and rdma. Not easy to find a high bandwidth switch that supports this stuff without breaking the bank.

Fibre Channel is a routed protocol invented specifically to connect block storage arrays to remote servers while making the block storage look locally attached to the OS. And it works REALLY REALLY well.
Post reply on HN