It's TCP vs. RPC All over Again

> the Internet seems to be missing a standard protocol for the request/response paradigm, with repeated attempts to force-fit TCP leading to inevitable mismatches

HTTP is that. QUIC is that over UDP.

We've discussed Ousterhout's paper about replacing TCP in the data center. Last I looked the answer was "not with what is described in the paper".

> I have never understood why the Internet has worked so persistently to adapt TCP in support of request/reply workloads instead of standardizing an RPC transport protocol to complement TCP.

> ...

> But my original question was to ask why that hasn’t happened. The only answer I can come up with is that judgment often reflects biases.

I can explain this. Some of it is bias, as TFA surmises, but there's a lot more to it.

First off, there is an Internet RPC protocol, and it's the ONC RPC protocol, the one that NFS uses. ONC RPC does run over TCP and UDP, which, yes, TFA doesn't seem to want.

Second point: adding new upper layer protocols (ULPs) like TCP and UDP and SCTP is fraught with middlebox and over-design problems, and you can look at SCTP for details.

The third point is that the ability to implement and iterate quickly means that having to wait for kernel support for the new thing is a non-starter. This is the strongest reason why you won't see a new RPC ULP happen quickly and why there has been little interest in one.

So between the middlebox issues and iteration latency issues the only way to do a new ULP is as a new binding of the RPC which must also have TCP and UDP bindings.

I.e., all request-response protocols we have pretty much have to run on top of TCP or UDP.

This fact of Internet life may suck, but that's just how it is. Making it go away will require lots of time. Developers don't have lots of time.

Fourth point: running a request response protocol over UDP is quite reasonable, especially in the datacenter.

Fifth point: building a new ULP will risk yielding the same over-design issues as SCTP all over again unless it's a new binding of another protocol.

I'm beating a dead horse here; let's switch to the "bias" issue.

RPC got a really bad rap in the 90s and 00s. The reason is that the RPC frameworks that came out of the 80s, and the more modern versions that followed them in the 90s and 00s, were all [mostly] synchronous. There's no reason for that: it was the easiest way to implement back then.

Another aspect of synchrony is that RPC == remote procedure call and comes from an era when it was fashionable to build things like distributed remote shared memory (which is a performance disaster). The idea that RPC is like a function call instantly evokes synchrony even though RPCs are not condemned to be synchronous. That suggestion leads to aversion.

Another reason that RPC frameworks got a back rap back then was the encoding systems used. It doesn't matter if the encoding system was trivial like XDR or complex like NDR or DER or PER or whatever -- the tooling for encoding rules from the 80s and 90s was just not remotely universal. As well we had the XML era (e.g., SOAP), which was very buzzword-rich and efficiency-poor.

That bad reputation is undeserved, really, but there it is. Nowadays RPC is making a comeback via gRPC and friends, but it shouldn't surprise that they typically run over HTTP.

Meanwhile the web folks iterated quickly. HTTP user-agent stacks quickly became async. HTTP APIs iterated from XML to JSON, and either way, with JavaScript and HTML, web apps got a great deal of flexibility and performed great by comparison to RPC apps.

Plus the web security model is atrocious mainly because of the most useful thing about HTML: pages can have cross-site links. Cross-site linking is so incredibly useful that it can't be done without. And now you need a fancy user-agent (browsers), so the web folks built one (browsers).

HTTP won out over RPC.

Why not use HTTP for everything, then? Well, HTTP/1.1 is... not efficient. H2 is much better, but still over TCP. H3/QUIC should be perfectly fine in the datacenter, and for some things on the web (and eventually maybe all things on the web?), and runs over UDP.

But HTTP is a request/response protocol. Just like any RPC!

Plus ça change, plus ça devient la même chose.

My argument: HTTP is the protocol you're looking for. And if ever we need a new ULP and are confident that we can deploy it, then we should design one for HTTP because a) that will satisfy the RPC need, b) it will tend to quash over-design instincts, c) it will have the same APIs/semantics as a protocol we're all already very familiar with, d) having one protocol with bindings to all applicable ULPs will help us get over the kernel support issues. (d) is especially appealing: you get to target one API and you get the best ULP locally supported.

We do have a widely-deployed Internet RPC protocol called HTTP, and you get to run it over TCP or UDP, and its semantics are stable when we add new ULP support. That's a pretty good outcome. Though maybe it requires getting over one's own biases :)

> That we have since turned HTTP into the Internet’s de facto RPC protocol (and then now realizing that it is suboptimal, are trying to optimize it by collapsing all the layers into the new QUIC protocol), is only a testament to how small a role technical rationale plays in what happens in industry.

I object to the characterization of HTTP as suboptimal for RPC. TFA is clearly referring to TCP as being suboptimal for RPC, but not really covering how UDP is suboptimal for RPC. And TFA does not discuss why NFS switched from using UDP to TCP, say.

It feels like TFA is just emoting that we ought to have a ULP specifically for pure RPC. That seems like a very biased position.

Suppose we built a ULP for RPC though. We could easily expect HTTP to get adapted to run over it. What does HTTP add that an RPC doesn't need? Mostly headers, which the RPC can just not use, and URI components, which the RPC can also set to the smallest possible values, say.

There's not that much light between RPC and HTTP. Both are request-response protocols. Accept HTTP as an RPC and you might be happy.

If you change the stack at TCP level...

- How many hardware network components would need replacing/upgrading to support a layer not expected to change?

- How many network + application software libraries would need to support parsing at packet level instead of byte level?

- How many OSI diagrams showing that high level byte and string based protocols run above the transport layer need to be re-written to say that's no longer the case.

- How many versions of RPC would need to be supported... just look at the numerous revisions of CORBA/RMI/SOAP(Document or RPC Style!)/REST/GRPC to see how many times you can re-invent (and misinterpret, break standards of) a wheel.

- Look how long IPv6 needed buy in and refinement, and then adoption. That was a necessary change with no alternative too!

There are many applications that shouldn't need TCP, such as those only doing short network hops or needing real time traffic with expected packet loss. They should use UDP but often don't. TCP is ubiquitous for the very reason that it is the chameleon mentioned in the article - TCP is not ideal, but it is convenient and well understood.

Consider how Lustre RPC works. It's mainly based on RDMA-capable protocols, though it has a TCP binding as well. For the TCP binding Lustre uses separate connections for RPC headers and bulk data, and even in the TCP case it's built around RDMA as a concept. RDMA essentially means that before you do a bulk I/O you negotiate acquisition of buffers for that bulk I/O, which imposes discipline on resource usage, which is pretty great for HPC and... less great on the public Internet. For writes this means an extra round trip at least once to set up the buffers you will use.

Now, the thing about RDMA is that it greatly benefits from HW NIC support -think InfiniBand-, but for that to be feasible the RDMA protocol needs to be its own ULP running over IP or -even better- a protocol at the same layer as IP. (If it's not run over IP, however, it can't be an Internet protocol.) This is where the real need for new protocols other than TCP or UDP should come in. And there are some such protocols, like RoCEv2.

It's totally possible to build an RDMA protocol over UDP that is performant. One might worry about the UDP header overhead, but it's minimal, and besides, you'll want cryptography, which will add more overhead for authentication tags. Don't allow fragmentation, meaning you'll need PMTUD, and you don't need your own length fields, so the only UDP overhead that could be avoided is the src/dst port numbers (4 bytes!)... which you might have anyways in a new ULP.

Do you mean the `select` call? libuv and friends can offer higher throughput than `select`.

Flatbuffers, Cap'n Proto, or SBE over RDMA.