281–290 of 329 posts
Earlier quoted context omitted.
> The ultimate dream protocol is one in which a sender just encodes bits in a certain way such that the receiver will get them, and puts them on the line without any handshaking or synchronization. This is a recipe for DDoS. Some handshaking is always necessary. You can minimize it, but you can't get rid of it.
> This is a recipe for DDoS. Inside a datacenter?
I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…
So storage doesn't use TCP, except for the protocols that are actually used, which do use TCP?
Earlier quoted context omitted.
not really. they conflate infiniband with RoCE which given they have different semantics on congestion control, I'd say is a bit of a whoopsey. if they are using RoCE, are they using DCB to avoid loss(well make it "lossless")? the paper implies otherwise.
For those who don’t know, RoCE is somewhat of a failure in the marketplace right now.
* https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet
> RDMA over Converged Ethernet (RoCE) is a network protocol that leverages Remote Direct Memory Access (RDMA) capabilities to accelerate communications between applications hosted on clusters of servers and storage arrays. RoCE incorporates the IBTA RDMA semantics to allow devices to perform direct memory-to-memory transfers at the application level without involving the host CPU. Both the transport processing and the memory translation and placement are performed by the hardware which enables lower latency, higher throughput, and better performance compared to software-based protocols.
* https://docs.nvidia.com/networking/pages/viewpage.action?pag...
If you're strictly operating inside a DC, presumably with minimal/fewer firewalls, could alternatives like SCTP and DCCP be an option?
* https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...
* https://en.wikipedia.org/wiki/Datagram_Congestion_Control_Pr...
Its easy to look at it and say it could be better because you have tunnel vision for your use case. However its also easy to forget that the protocol literally has a billion edge cases.
Why do you think your NIC has a new driver update every couple of weeks even though they have been running the same protocols for the last 50 years?
Earlier quoted context omitted.
Yes you can. Just offer a better product, and people will buy it instead of the old or bad product. Better yet, make the new product backwards compatible, and fewer people will have qualms about forking out for it. Better yet, do an aggressive takeover, like Microsoft did, and just force the entire industry to adopt your stuff...
Great! When do you think you'll have it done?
Earlier quoted context omitted.
> Google hasn't used TCP in the datacenter for years. That's absolutely false. I don't have any sources except for having worked at Google from 2013-2022, but it's not like you quoted any sources either, so... There's a reason why Google is still releasing stuff like TCP BBR (2017).
Well, the first link googling 'google datacenter hardware' is google's article on how they don't use standard TCP hardware or software in their datacenters. But I guess that was too much to ask...
If you link to the article in question (and relevant quotes) I'm happy to try and clarify your misunderstanding.
I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would mov…
To be fair the paper talks a bit about how TCP makes multithreading slower compared to a message based system.
Earlier quoted context omitted.
After having lived through Amazon's early (pre-2003ish) UDP-based networking I got a laugh around 2006-ish or so reading about how facebook was into UDP. I assume there are people who worked there who still have the scars.
Do you have any specific problems you can elaborate with the UDP ? UDP used successfully many places.
The really key point there is probably the size of the responses, it wasn't just tiny atomic bits of stock information.
At one point as a system engineer I actually had to bump up the size of UDP socket data that the kernel would allowed to be sent across the entire production set of servers. SWEs were really hammering on UDP hard (the platform framework was sort of "sold" as being better than TCP though which doesn't have those kinds of limits).
The result was that one Christmas the traffic scaled up to the point that the switch buffers were routinely overflowing all the time. There was no slow start in UDP so the large payloads the SWEs were sending would go out as fast as the NICs could send them, which resulted in filling up packet buffers in the 6509s (Sup 720s I think at the time? Whatever it was the network engineers had already upgraded to whatever was Cisco's latest and greatest at the time and had tuned the switch buffers).
What made it even more fun was that as packets were dropped on the multicast routes the unicast replies created a bit of a bandwidth-amplification-attack. Then eventually the switch buffers started dropping IGMP packets, and if you drop enough of those in a row then IGMP sniffing fails and the multicast routes themselves start getting torn down. Now you get "packet loss" on one of the destination nodes which is complete. Then when it eventually rejoins it has fallen far behind all the peers (causing a bunch of issues when it was out of synch to begin with though) and then it requests more unicast messages to get caught up, creating even more of a flood of rapidly-sent UDP.
What I wound up doing is writing scripts to log into all the core switches and dump out the multicast tables and convert the IGMP snooped routes into static routes and reapply them. That let the multicast network grow as the site had to scale for Christmas, but kept all the routes in place and avoided the IGMP route flapping.
But even with that band-aid it still didn't work well and there was still high congestion and packet loss across the core switches. There were also problems with the CPU on the switches and Amazon drafted an extension to how multicast routing was done and got Cisco to implement it ("S,* routing" IDK if that's right its been 20 years). And it was a good job that the Network Engineers had ripped out spanning tree and gone L3 entirely since the packet loss and CPU congestion would have caused spanning tree to flap which would have amplified all the congestion issues. Eventually Tibco RVD was ripped out and a TCP-based gossip-based-clustering protocol was put into place.
So if you use UDP based stuff the datapackets need to be small, or else you need to throttle the senders somehow, and you need to not care about reliability. For stock ticker information it might work well, and for multimedia streaming where the protocol layer above it does slow start and congestion control. I suspect that if you dug up the network engineer responsible for those networks though that they could tell you stories about packet loss. If UDP works well at your company my suspicion is that you've either got a protocol sitting on top of UDP which implements at least half of what TCP offers, and/or you've got an overworked network engineer trying to keep it all together, and/or you just haven't scaled enough yet. I also wouldn't be too surprised if some wall street firms have switched to RDMA-over-Infiniband or something like that with link-layer and end-to-end credit-based based flow control[*] (as this paper points out, though, RMDA has issues itself and doesn't meet all the criteria for a TCP-replacement, but that would at least stop the packet loss issues due to buffers overflowing).
QUIC is a good example of what you need to do in order to use UDP (Section 4 of RFC 9000 is all about Flow Control to prevent fast senders from DoS'ing your network switches). But for the average HN/reddit reader who reads something about how TCP is awful and has the "showerthought" of wondering about why everyone doesn't just switch to UDP in the datacenter, they're missing a massive problem in that Ethernet has no flow control and just promiscuously drops packets everywhere, so if you thoughtless slap UDP on top of that your datacenter will absolutely have a meltdown. You need to use something like QUIC at a minimum.
And buried in what I wrote above is an observation that UDP multicast doesn't really solve reliable delivery across multiple servers and failover of streams that you'd like to be able to see, that's another solution which is simple and wrong (and which it looks like Homa is trying to address).
[*] On second thought they probably massively overprovision their network since mostly they just care in the extreme about latency at the expense of everything else (which is a very unusual use case).