121–130 of 329 posts
I was about to rant that Prof. Ousterhout should just deploy some of his students and get that transport - RPC integration done and prove out his point. But, then I tried to look for it first and found this:
https://www.usenix.org/system/files/atc21-ousterhout.pdf
Has anybody tried it in an actual data-center?
Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer probl…
This isn't just a datacenter problem. Every single network protocol I've ever created or implemented is message based, not stream based. Every messaging system. Every video game. Every RPC transport.
But, because we can't have nice things, message framing has to be re-implemented on top of TCP in a different, custom way by every single protocol. I've basically got message framing-over-TCP in muscle memory at this point, in each of the variants you commonly see.
The only kinda-sorta exceptions I know about are HTTP/1.1 and telnet. But HTTP/1.1 is still a message oriented protocol; just with file-sized messages. (And even this stops being true with http2 anyway).
In my opinion, the real problem is the idea that "everything is a file". Byte streams aren't a very useful abstraction. "Everything is a stream of messages" would be a far better base metaphor for computing.
Earlier quoted context omitted.
> 50 years of clock time and thousands of years of engineer time It's not just the size of the investment, it's that it's the protocol everyone uses to talk to other people's machines, and you can't upgrade or replace other people's machines.
Yes you can. Just offer a better product, and people will buy it instead of the old or bad product. Better yet, make the new product backwards compatible, and fewer people will have qualms about forking out for it. Better yet, do an aggressive takeover, like Microsoft did, and just force the entire industry to adopt your stuff...
Earlier quoted context omitted.
> corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. This is ridiculous. Hyperscalars see an immediate ROI from efficiency/reliability improvements and actively invest in TCP alternatives all of the time. It's just really hard. Networking companies see an ability to differentiate their products from their peers and work on this kind of thing as well…
> 50 years of clock time and thousands of years of engineer time It's not just the size of the investment, it's that it's the protocol everyone uses to talk to other people's machines, and you can't upgrade or replace other people's machines.
To elaborate on fairness, dynamic traffic control of all flows within a DC while maintaining high utilization is roughly impossible. You can get really close to this by picking your battles wisely (i.e. solid demand control for data warehouse workloads), but you'll always end up counting on individual flows to react appropriately to loss. They need to back off enough to make room for others without tanking their own throughput.
The people who design and implement these algorithms are definitely geniuses, but even they rely on TONS of empirical evidence to narrow parameters to what's appropriate. Of the Kernel Networking people I've worked with, Lawrence Brakmo had the most sophisticated network testing harness I've seen. Even then, you don't really know if it works (and can't finish tuning it) until you run it in production.
Running novel congestion control algorithms in production at a sufficient scale to figure out whether or not they're working appropriately is a great way to kill your network, so we end up conducting the equivalent of CCA drug testing to roll it out slowly and safely.
The end result of all of this is that it's really hard to solve the "arbitrary connections sharing arbitrary network topologies with high utilization" problem quickly enough for it ever to look like a breakthrough rather than just steady progress.
It's also worth noting that it's usually easiest to prove performance, so you'll see a lot of excitement about performance benchmarks from people who don't yet know what they're about to learn about networking. We were very much in this camp at Facebook when we were all-in on memcache-over-udp, and we later abandoned it completely.
> The data model for TCP is a stream of bytes. However, this is not the right data model for most datacenter applications. Datacenter applications typically exchange discrete messages to implement remote procedure calls This isn't just a datacenter problem. Every single network protocol I've ever created or implemented is message based, not stream based. Every messaging system. Every video game. Every RPC transport.…
No, HTTP/2 and QUIC do not change the semantics of HTTP.
Also, you can have endless streams with HTTP/1.1: just use chunked encoding to POST/PUT and use Range: bytes=0- and chunked encoding for GET and chunked encoding for POST response bodies. In HTTP/2 there's only the equivalent of chunked encoding -- there's no definite content length in HTTP/2.
"We hypothesize that flow-consistent routing is responsible for virtually all of the congestion that occurs in the core of datacenter networks". Flow-consistent routing is the constraint that packets for a given TCP 4-tuple get routed through the same network path, rather than balanced across all viable paths; locking a flow to a particular path makes it unlikely that segments will be received out of order on the des…
All that is needed for congestion is for two large flows
to hash to the same intermediate link; this hot spot will persist
for the life of the flows and cause delays for any other
messages that also pass over the affected link.
The ultimate dream protocol is one in which a sender just encodes bits in a certain way such that the receiver will get them, and puts them on the line without any handshaking or synchronization. I don’t think this is impossible. The space of orthogonal codes across time and frequency could be chosen to be practically infinite, therefore, any random selection of two such codes would look like white noise to each othe…
This is a recipe for DDoS.
Some handshaking is always necessary. You can minimize it, but you can't get rid of it.