gRPC: The Bad Parts

Yeah my memory of this (which is admittedly fuzzy) is that a bunch of business things happened around 2012-2015, which led to these external software libraries / products that are "arguably inferior rewrites" and "not what Google actually uses"

I think 2012 is when Larry became CEO (again), and 2015 is when the "Alphabet" re-org / re-naming happened.

1. Larry Page was generally unhappy with the direction and execution of the company, so he became CEO. (Schmidt would never be CEO again)

2. VP Bill Coughran was shown the door (my interpretation, which is kind of like Eric Schmidt being shown the door). For my entire time there he had managed the software systems -- basically everything in google3, or everything important there

3. Urs Hoezle took over everything in technical infrastructure. I think he had previously been focused on hardware platforms and maybe SRE; now he was in charge of software too.

Urs sorta combined this "rewrite google3" thing with the "cloud" thing. To me there was always a tenuous connection there, at least technically. I can see why it made sense from a business perspective

---

Basically Larry was unhappy with google3 because the company wasn't shipping fast enough, e.g. compared to Facebook. It was perceived as mired in technical debt and processes (which IMO was essentially true, and maybe inevitable given how fast the company had grown for ~8 years)

And I think they were also looking over their shoulders at AWS, which I think by then had become "clearly important".

Why don't we have an AWS thing? At some point GCE was kind of a small project in Seattle, and then it became more important when AWS became big.

Anyone remember when Urs declared that google3 was deprecated and everything was going to be written on top of cloud in 12 to 18 months? (what he said was perhaps open to interpretation -- I think he purposely said something really ambitious to get everyone fired up)

So there was this shift to "externalize" infrastructure, make it a real product. Not just have internal customers, but external ones too.

---

So I think what you said is accurate, and I think that is the business context where the "arguably inferior rewrites" came from

- Kubernetes is worse in many ways than Borg [1]

- gRPC (I haven't used it) is apparently worse in many ways than Stubby, etc.

I'd be interested if anyone has different memories ...

---

[1] although I spent some time reading Borg source code, and e.g. compared to say the D storage server, which was also running on every node, it was in bad shape, and inefficient. There are probably ways that K8s is better, etc.

My main beef is the unimaginable complexity of running K8s on top of GCE on top of Borg -- i.e. 3 control planes stacked on top of each other ...

The technical reason is that they break the end-to-end principle. In computer networking in general, it is a desirable property for an opaque message to remain opaque, and agnostic to every server and system that it passes through. This is part of what makes TCP/IP so powerful: you can send an IP packet through Ethernet or Token Ring or Wi-Fi or carrier pigeon, and it doesn't matter, it's just a bunch of bytes that get passed through by the network topology.

Protobufs generally respect this property, but required fields break it. If you have a field that is marked 'required' and then a message omitting it passes through a server with that schema, the whole message will fail to parse. Even if the schema definition on both the sender and the recipient omits the field entirely.

Consider what happens when you add a new required field to a protobuf message, which might be embedded deep in a hierarchy, and then send it through a network of heterogenous binaries. You make the change in your source repository and expect that everything in your distributed system will instantly reflect that reality. However, binaries get pushed at different times. The sender may not have picked up the new schema, and so doesn't know to populate it. The recipient may not have picked up the new schema, and so doesn't expect to read it. Some message-bus middleware did pick up the new schema, and so the containing message which embeds your new field fails to parse, and the middleware binary crashes with an assertion failure, bringing down a lot of unrelated services too.

To get feature parity, you still need an IDL to generate types/classes for multiple languages. You could use JSON Schema for that.

Websockets do not follow a request/reply semantics, so you'd have to write that yourself. I'd prefer not to write my own RPC protocol on top of websockets. That said, I'm sure there are some off the shelf frameworks out there, but do they have the same cross-language compatibility as protobuf + gRPC? I don't think "just use JSON with websockets" is such a simple suggestion.

Of course, gRPC does have some of its own problems. The in-browser support is not great (non-existent without a compatibility layer?) last time I checked.

Java is doing quite well.

What an idiotic comment.