Buf raises $93M to deprecate REST/JSON

When I worked at Google I sat down one day across from one of the gRPC engineering leads who was talking about the things they were doing for the then current generation of gRPC. I asked if I could ask some questions about it and they agreed and then dissected their design in half a dozen ways that would fail in both non-important but irritating ways, and in critical ways at scale. They were amazed at I had thought about this topic so deeply as it was all “state of the art” and I was, nominally, “old.” I pointed out that I had been the ONC RPC architect at Sun in the ‘80s during the first RPC wars and while the implementations had change, fundamentally messaging as a form of procedure call has some fundamentally bad properties. These challenges manifest in all aspects of RPC, from marshaling data, to rendezvousing, to delivery reliability and guarantees. Andy Birrell at DEC SRC and Leslie Lamport had written dozens of papers looking at these challenges in small systems and large. There was literally decades of solid research that the engineer in front of me at the cafeteria that day was re-discovering from first principles.

RPC protocols from Sun, DEC, Microsoft, OSI, SOAP, the IETF, and the Open Group have run at this problem again and come up with different solutions that each have their own set of warts. Good for some things, not great for others. But at this point there are enough options at this point.

What is missing from Buf’s material is what I might call the “Chesterson’s fence” material that dives into why all of these previous versions were insufficient and how their new version of gRPC will solve all those problems without adding new wrinkles.

I think it is great that they are trying to improve the state of the art, I would feel better about it if they also demonstrated they understood what had come before.

- C++: https://github.com/grpc/grpc/blob/master/bazel/cc_grpc_libra...

- Python: https://github.com/grpc/grpc/blob/master/bazel/python_rules....

- ObjC: https://github.com/grpc/grpc/blob/master/bazel/objc_grpc_lib...

- Java: https://github.com/grpc/grpc-java/blob/master/java_grpc_libr...

- Go (different semantics than all of the other): https://github.com/bazelbuild/rules_go/blob/master/proto/def...

But there's also no real cohesion within the community. The biggest effort to date has been in https://github.com/stackb/rules_proto which integrates with gazelle.

tl;dr: Low alignment results in diverging implementations that are complicated to understand for newcomers. Buff's approach is much more appealing as it's a "this is the one way to do the right thing" and having it just work by detecting `proto_library` and doing all of the linting/registry stuff automagically in CI would be fantastic.

Maybe I'm just another clueless millennial developer who doesn't understand the history of the 80's or whatever, but I've never been able to understand this claim that RPC is broken. There's a lot of assertions that everyone knows RPC was broken because smart people in the 80's said so but... no one has ever been able to give me a concrete reason why.

RPC, at least as I've always known it, really just boils down to request/response protocols. You send a request, you get a response. While this is admittedly not the only possible networking pattern, it is the dominant one across almost all distributed systems I've worked with. HTTP itself is request/response -- it's basically the same thing.

All gRPC and Protobuf are doing that is different from HTTP is they're using a binary protocol based on explicitly-defined schemas. The protobuf compiler will take your schemas and generate convenient framework code for you, so you don't have to waste your time on boilerplate HTTP and parsing. And the binary encoding is faster and more compact than text-based encoding like JSON or XML. But this is all convenience and optimization, not fundamentally different on a conceptual level.

Neither HTTP nor RPC protocols have ever pretended to solve higher-level distributed systems concerns of fault tolerance, network partitions, reliable delivery, etc. Those are things you build on top. You need a basic mechanism to send messages before you can do any of that.

What, exactly, is the magical non-RPC approach of the 80's that we're all missing? Can you explain the alternative?

EDIT: Also, like, the entirety of Google is built out of services that RPC to each other, but the 80's called and said that's wrong? How am I supposed to take this seriously?

For those who still want / need binary protocols and schemas, look at FlatBuffers or Cap'n Proto instead. At least they are capable of representing domain structures properly.