Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's time to replace TCP in the datacenter (2023)

arxiv.org

Post reply on HN
Print view View original

121–130 of 160 posts

Re: It's time to replace TCP in the datacenter (2023)

#121

Post by ironhaven »

pif wrote: post #49

> Although Homa is not API-compatible with TCP, IPv6 anyone? People must start to understand that "Because this is the way it is" is a valid, actually extremely valid, answer to any question like "Why don't we just switch technology A with technology B?" Despite all the shortcomings of the old technology, and the advantages of the new one, inertia _is_ a factor, and you must accept that most users will simply even re…

But tcp/ipv6 is API compatible with tcp/ipv4? You can even accept ipv4 connections to a ipv6 listening socket if you have a ipv4 address assigned. The issue is ipv4 is not forward binary compatible with ipv6 because you can't fit more than 2^32 addresses in a ipv4 packet.

But yeah if you are a large bloated enterprise like amazon or microsoft that owns large amounts of ipv4 address space and expensive ipv4 routing equipment there is not a ton of value in switching

Re: It's time to replace TCP in the datacenter (2023)

#122

Post by jabl »

tonetegeatinst wrote: post #108
UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

Fiber is attractive. As someone who wants to upgrade to fiber, the main barrier to entry is the cost of switches and a router. Granted I'm also trying to find a switch that supports ROCm and rdma. Not easy to find a high bandwidth switch that supports this stuff without breaking the bank.

Fiber, as in Fibre Channel (FC, https://en.wikipedia.org/wiki/Fibre_Channel ), not fiber as in "optical fiber" instead of copper cabling.

Re: It's time to replace TCP in the datacenter (2023)

#123

Post by skeptrune »

stego-tech wrote: post #89

As others have already hit upon, the problem forever lies in standardization of whatever is intended to replace TCP in the data center, or the lack thereof. You’re basically looking for a protocol supported in hardware from endpoint to endpoint, including in firewalls, switches, routers, load balancers, traffic shapers, proxies, etcetera - a very tall order indeed. Then, to add to that very expensive list of criteria…

> Even if they do (e.g., Google with QUIC), the broad vibe I get is that folks aren’t likely to trust those offerings as lacking in ulterior motives.

It's pretty unfortunate that we've landed here. Hordes of venture-backed companies building shareware-like software with an "open source" label has done some severe damage.

Re: It's time to replace TCP in the datacenter (2023)

#124

Post by michaelt »

andrewflnr wrote: post #92
michaelt wrote: post #76

Earlier quoted context omitted.

When you are an outsider it's wise to take such claims with a grain of salt, because as the "secret" made its way to you, recounted by one person to another to another, there might have been an exaggeration, over-simplification or misunderstanding. It's easy to imagine how, in the hands of tech journalists and youtubers optimising for clicks, "Google likes QUIC" and "Some ML clusters use infiniband" could get distort…

True as far as it goes, but "they didn't actually do it" is a different story from "they did it secretly". The two claims exclude each other, so you can't really compare them in the context of "credit".

Allow me to rephrase.

I think the statement is not true, in a literal sense. I do not think there are multiple FAANG companies with data centres where TCP has been entirely eliminated.

But the statement is ambiguous enough you could come up with true interpretations, if you diluted it to the point of meaninglessness.

Therefore, without a clear public statement of what is being claimed, it's very difficult for me to be impressed.

Re: It's time to replace TCP in the datacenter (2023)

#125

Post by wmf »

tonetegeatinst wrote: post #107

Many I am misunderstanding something about the issue, but isn't DCTCP a standard? See the rfc here: https://www.rfc-editor.org/rfc/rfc8257 The DCTCP seems like its not a silver bullet to the issue, but it does seem to address some of the pitfalls of TCP in a HPC or data center environment. Iv even spun up some vm's and used some old hardware to play with it to see how smooth it is and what hurdles might exist, but th…

Homa has much lower latency than DCTCP.

Re: It's time to replace TCP in the datacenter (2023)

#126

Post by mananaysiempre »

Sebb767 wrote: post #60
ksec wrote: post #52

Earlier quoted context omitted.

I wonder if there are any work on making something similar ( conceptually ) to TCP, super / sub set of TCP while offering 50-80% benefits of HOMA. I guess I am old. Everytime I see new tech that wants to be hyped, completely throw out everything that is widely supported and working for 80-90% of uses cases, not battle tested and may be conceptually complex I will simply pass.

If you have a sufficiently stable network and/or known failure cases, you can already tune TCP quite a bit with nodelay, large congestion windows etc.. There's also QUIC, which basically is a modern implementation of TCP on top of UDP (with some trade-offs chosen with HTTP in mind). Once you stray too far, you'll loose the ability to use off-the-shelve hardware, though, at which point you'll quickly hit the point of…

One issue with QUIC in e.g. C is how heavyweight it feels to use compared to garden-variety BSD sockets (and that’s already not the most ergonomic of APIs). I haven’t encountered a QUIC library that didn’t feel like it would absolutely dominate a simple application in both code size and API-design pressure. Of course, for large applications that’s less relevant, but the result is that switching to QUIC gets perceived as a Big Deal, a step for when you’re doing Serious Stuff. That’s not ideal.

I’d love to just play with QUIC a bit because it’s pretty neat, but I always get distracted by this problem and end up reading the RFCs, which so far I haven’t had the patience to get through.

Re: It's time to replace TCP in the datacenter (2023)

#127

Post by FaceValuable »

wutwutwat wrote: post #119
shawn_w wrote: post #116

Earlier quoted context omitted.

Unix domain stream sockets do not use tcp. Nor do unix datagram sockets use udp. They're much simpler.

>The type parameter should be one of two common socket types: stream or datagram.[10] A third socket type is available for experimental design: raw. > SOCK_STREAM will create a stream socket. A stream socket provides a reliable, bidirectional, and connection-oriented communication channel between two processes. Data are carried using the Transmission Control Protocol (TCP). > SOCK_DGRAM will create a datagram socket.…

Interesting! The Wikipedia article is quite wrong here. SOCK_STREAM certainly doesn’t imply TCP in all cases. I see the source is the Linux Programming Interface book; quite likely someone’s interpretation of that chapter was just wrong when they wrote this article. It is a subtle topic.

Re: It's time to replace TCP in the datacenter (2023)

#128

Post by Sylamore »

UltraSane wrote: post #12

I wonder why Fibre Channel isn't used as a replacement for TCP in the datacenter. It is a very robust L3 protocol. It was designed to connect block storage devices to servers while making the OS think they are directly connected. OSs do NOT tolerate dropped data when reading and writing to block devices and so Fibre Channel has a extremely robust Token Bucket algorithm. The algo prevents congestion by allowing receiv…

InfiniBand would make more sense than Fibre Channel

Re: It's time to replace TCP in the datacenter (2023)

#129

Post by getcrunk »

skeptrune wrote: post #123
stego-tech wrote: post #89

As others have already hit upon, the problem forever lies in standardization of whatever is intended to replace TCP in the data center, or the lack thereof. You’re basically looking for a protocol supported in hardware from endpoint to endpoint, including in firewalls, switches, routers, load balancers, traffic shapers, proxies, etcetera - a very tall order indeed. Then, to add to that very expensive list of criteria…

> Even if they do (e.g., Google with QUIC), the broad vibe I get is that folks aren’t likely to trust those offerings as lacking in ulterior motives. It's pretty unfortunate that we've landed here. Hordes of venture-backed companies building shareware-like software with an "open source" label has done some severe damage.

Quic is an attack on network based filtering (ad blockers) similar to doH. There likely are convenient ulterior motives.

Re: It's time to replace TCP in the datacenter (2023)

#130

Post by bmitc »

j16sdiz wrote: post #29
bmitc wrote: post #24

Earlier quoted context omitted.

Yes, I know that WebSockets layer over TCP/IP. But that both misses the point and is part of the point. The reason that I ask is that WebSockets seem to almost always be used in the context of web applications. TCP/IP still seems to dominate control communications between hardware. But why not WebSockets? Almost everyone ends up building a message framing protocol on top of TCP/IP, so why not just use WebSockets whic…

WebSocket is fairly inefficient protocol. and it needs to deal with the upgrade from HTTP. and you still need to implement you app specific protocol. This is adding complexity without additional benefit It make sense only if you have an websocket based stack and don't want to maintain a second protocol.

> WebSocket is fairly inefficient protocol

In what way?

> This is adding complexity without additional benefit

I'm not sure that's a given. For example, WebSockets actually implement a message protocol. You have gauarantees that you sent or received the whole message. That may not be the case for TCP/IP, which is a byte streaming protocol.

Post reply on HN