251–260 of 329 posts
Earlier quoted context omitted.
> The ultimate dream protocol is one in which a sender just encodes bits in a certain way such that the receiver will get them, and puts them on the line without any handshaking or synchronization. This is a recipe for DDoS. Some handshaking is always necessary. You can minimize it, but you can't get rid of it.
> This is a recipe for DDoS. Inside a datacenter?
EDIT: But you know, UDP meets your bill.
Earlier quoted context omitted.
The reality of today's large datacenters is that almost all of them have almost all of their traffic on TCP unless the owners of the datacenter have made a conscious effort to not use TCP. The highest-traffic applications, usually databases and storage systems, pretty much all use TCP unless you are buying a purpose-built HPC scale-out storage system (like a Lustre cluster). Most people who build a datacenter today u…
Fibre channel is dead, long live fibre channel. I agree a lot of things are on TCP, but I don't think its a massive problem, unless you are running close to the limit of your core network. And one solution to that is to upgrade your core network.... Failing that, implementing some load balancing/partitioning systems to make sure data-processing affinity is best matched. This the better solution, because it yields oth…
23 years ago I sat in a meeting with Sun, Intel, Mellanox, and 1 or 2 others. In that meeting we discussed putting an RDMA interface on individual hard drives, trays of RAM, CPUs, and other more exotic devices (like battery backed RAM, no conventional SSD in those days of course). You’d install RAM 1 42U rack at a time, disks likewise, CPUs in another rack and so on. All partitioned, controlled, managed, and of cours…
I get the issue with TCP, but I'm not sure about Homa... e.g., why not UDP with some homa-like semantics on top? That might really ease the "Getting there from here" issue. In fact, what's actually being suggested is for applications to replace calls to TCP-based APIs with calls to gRPC APIs (or other high-level RCP APIs), where the transport layer becomes an implementation detail. Fair enough, but this is a very rou…
Skimming over the paper, I think the magic of Homa is in its RPC calls and its short-lived connections. When they're handled at layer 3 and 4, they can provide a significant hint to switches, routers and hosts regarding prioritization and congestion control. If you agree upon the prioritization algorithms as part of the protocol, then both sending and receiving hardware can coordinate much more easily.
If we instead just implemented something like Homa on top of UDP, it would basically mean that the top of the OSI stack would have to somehow inform the lower layers of the stack about these "sessions". You'd also have to hope that 3rd party peers decide to implement the hints in the same way. This would result in much more complexity.
Google hasn't used TCP in the datacenter for years. What they use I don't know. But it's even custom switches with custom chips. My son did work in graduate school for a clean-slate network implementation of a network for the datacenter. Maybe Google, I don't remember. One issue I remember they addressed was, scheduling bandwidth for VM migration within their datacenter cloud. See, some customer reserves a 'machine'…
https://www.virustotal.com/gui/url/43f33fe70cb4ef9fcc2370460...
Earlier quoted context omitted.
I suspect there are a couple of contributors. TCP is prevalent on the internet, so you need a fairly strong motivation and benefits to adopt a second protocol. A lot of engineering also doesn't get underlying networking, so one of the successes of TCP is it's a file descriptor that you either write to or read from and magic makes it come out the other side. I've seen tech leadership on networking centric products kno…
How does your theory that the failure of SCTP is because a) people don’t understand networking and b) tcp eats up all the development oxygen explain QUIC? I’m also not sure what you mean but DCs within a major cloud provider are majority AFAIK running truly isolated networks interconnected directly with fiber. If you haven’t yet, I would recommend reading the very original QUIC paper. It was extremely astute and show…
I think this is the motivation side of the argument. SCTP doesn't provide any advantage internally for most use cases, as I outlined my thoughts on the basis above. QUIC on the other hand is an attempt to solve a completely different set of problems, and is getting the engineering dollars to deploy because where latency and internet comes into play, there is a strong motivation to be faster. And it also becomes more of an upgrade path.
> I’m also not sure what you mean but DCs within a major cloud provider are majority AFAIK running truly isolated networks interconnected directly with fiber.
Sorry about being unclear, I typed the out pretty quickly. One of the main factors that drove Telecom to create and adopt SCTP, is the way telecoms like to interconnect with eachother. For signaling traffic (message like I want to setup a new phone call), the telco's like to set up multiple independent connections. So with SCTP, they want multi-path support, where each server advertises a list of IP addresses for the connection. So between two telco's, you have a dedicated non-internet connection A, and a diverse network B. Equipment that communicates on these networks is then physically plugged into both networks. This creates a need for a protocol that understand this, and when a failure occurs in transmitting on the A network, retransmission occurs on the B network. The idea is these are diverse networks, nothin can really interact with both at the same time (that's the theory, in practice there be stories).
Where this maps to data center networks, is to my knowledge most data center networks are not designed into an A and B network for diversity. Where you would have to use multipath TCP or SCTP. And if you want to use both together, you're going to design the network to support all the failovers and redundancy to deliver TCP.
So that's what I was trying to get at, where there is a big adoption driver and protocol complexity is on the multi-path support, which to fully utilize requires additional engineering effort in the data center.
A few years ago (when QUIC was coming out) I was developing the theory of a new transport/encryption/authentication protocol. The focus was as much on transport as in the built-in federated authentication. There was not much interest in the field and I had a lot of the theory and formal proofs, but no implementation. This month I found a cofounder and we are reordering a lot of the information and presentation, we sh…
The question would have to be "what does your idea/project offer that NATS doesn't already offer?"
I have no affiliation with NATS, but I wish that people were paying more attention to it. It solves a lot of problems people have.
> The data model for TCP is a stream of bytes. However, this is not the right data model for most datacenter applications. Datacenter applications typically exchange discrete messages to implement remote procedure calls This isn't just a datacenter problem. Every single network protocol I've ever created or implemented is message based, not stream based. Every messaging system. Every video game. Every RPC transport.…
Files are just an indexed list of bytes that can represent anything. Think of them as objects.
> Byte streams aren't a very useful abstraction. "Everything is a stream of messages" would be a far better base metaphor for computing.
I don't understand this. A stream of messages is a stream of bytes. Its bytes all the way down.