21–30 of 37 posts
Things I've learned using gRPC for ~10 years: 1. I really like gRPC and protos for the codegen capabilities. APIs I've worked on that use gRPC have always been really easy to extend. I am always tempted to hand-roll http.Handle("/foo", func(w http.ResponseWriter, req *http.Request) { ... }), but gRPC is even easier than this. 2. grpc-web never worked well for me. It's hard to debug; in the browser's inspector you jus…
Then web browsers never implement any of those capabilities.
gRPC-web's roadmap is still a huge pile of workarounds they intend to build. These shouldn't be necessary! https://github.com/grpc/grpc-web/blob/master/doc/roadmap.md
Instead of giving us http-push, everyone said, oh, we haven't figured out how to use it for content delivery well. And we've never ever let anyone else use it for anything. So to got canned.
Http-trailers also seems to not have support, afaik.
Why the web pours so much into HTML, js, CSS, but utterly neglects http, to the degree where grpc-web will probably end up tunneling http-over-webtransport is so cursed.
Things I've learned using gRPC for ~10 years: 1. I really like gRPC and protos for the codegen capabilities. APIs I've worked on that use gRPC have always been really easy to extend. I am always tempted to hand-roll http.Handle("/foo", func(w http.ResponseWriter, req *http.Request) { ... }), but gRPC is even easier than this. 2. grpc-web never worked well for me. It's hard to debug; in the browser's inspector you jus…
I love protobufs as a type-safe way of defining messages and providing auto-generated clients across languages. I can't stand gRPC. It's such a Google-developed product and protocol that trying to use it in a simpler system (e.g. everyone else) is frustrating at best, and infuriating at worst. Everything is custom and different than what you're expecting to deal with when at its core, it is still just HTTP. Something…
Yea, and the code gen (for Go at least) very clearly assumes you're using a monorepo and how dare you think of doing anything else you monster. E.g. there's a type registry, which means you can't ever have the same proto type compiled by two different configs (it'll panic at import time). In a monorepo that's (potentially) fine, but for the rest of the world it means libraries can't embed the generated code that they…
We never generated go code from it, but it took a bit of fine tuning to get generated code that felt at least somewhat ergonomic for Ruby and Typescript. It usually involved using some language specific alternative to protoc for that language because the code generated by protoc itself was practically unreadable. IIRC in the case of Typescript I had to write a script that messed around with the directory structure so you could use sensible import paths and aliases, because TS itself wasn't discovering them automatically without it.
That's stuff you can work with and solve technically. Initial faff but it's one and done. The worst problem I had with it was protobuf3 stating every field is optional by default, and the company I worked at basically developed a custom schema registry setup that declared every field as required by default, with a custom type to mark a field as optional. It turned literally every modification to a protobuf definition into a breaking change and, what's worse, it wasn't done end to end and the failures were silent, so you'd end up with missing data everywhere without knowing it for weeks.
If you're picking communication protocols for your API endpoints, you're doing it wrong. The ideal API supports `/api/call.json` and `/api/call.csv` and `/api/call.arrow` as well as `/api/call.grpc`, and the only thing that differs between these is the serializer (which is standardized and very well tested). If the App is buggy, I want to (1) check what API calls it's making (2) be able to run those API calls myself,…
Oh, I'm also supporting streaming. Will `/api/call.rtmp` work for me?
What's worse, I even allow voice chat and video calls! I guess I can just do `/api/call.webrtc`?
I also forget that I serve a lot of P2P traffic, so I guess I can just add `api/call.udp` too.
The problem I have now is, how do I serve this over HTTPS? I can't pick the protocol because that'd be doing it wrong. Do I serve extra APIs like `call.rtmps`, `call.webrtcs`, `call.udps`?
I haven't even told you how fucking awkward it is to constantly enter my login details to work with my git repository and `github.com/my/repo.ssh` just gives me a 404.
APIs are a lot more than just content negotiation; the protocol is a necessary part of it and it just so happens with the web (not the internet) that the protocol is HTTP(S) over TCP and advances in technology have continued to stress its utility.
Earlier quoted context omitted.
Yea, and the code gen (for Go at least) very clearly assumes you're using a monorepo and how dare you think of doing anything else you monster. E.g. there's a type registry, which means you can't ever have the same proto type compiled by two different configs (it'll panic at import time). In a monorepo that's (potentially) fine, but for the rest of the world it means libraries can't embed the generated code that they…
I used protobuf extensively with Kafka and I remember having to be quite particular about how the proto files/packages were arranged so as to avoid naming and versioning conflicts. We never generated go code from it, but it took a bit of fine tuning to get generated code that felt at least somewhat ergonomic for Ruby and Typescript. It usually involved using some language specific alternative to protoc for that langu…
This is the main reason I think protobuf's "zero values are simply not communicated" is fundamentally wrong. Missing data is one of the easiest flaws to miss, and it tends to cause problems far away from the source of the flaw, in both time and space, which makes it extremely hard to notice and fix.
I get the arguments in its favor. I get the arguments in favor of "everything is optional by default". But presence is utterly critical in detecting flaws like this, and it can't always be addressed in a backwards-compatible way by application code. E.g. in proto's case, it's not possible because that data does not exist, and adding it would change the binary data. Even binary-compatible workarounds like "add a field with a presence fieldset" aren't usable because that unrecognized field will be silently ignored by older consumers, so you're right back to where you started.
It needs to exist from day 1 or you're shooting your users in the feet.
I haven't done any gRPC programming, but from what I understand it's a modern re-imagining of XML Web Services: you describe the contract and can generate client and server code based on it for different programming languages. Early on in my career I worked on several projects that used WS-* "stack", and generally it was a very good experience. Once we had a project that was split between two subcontractor teams and…
A contemporaneous competitor to Sun RPC was DCE/RPC (https://en.wikipedia.org/wiki/DCE/RPC), which I think Microsoft's SMB protocol was based upon, albeit in Microsoft's trademark manner--embrace, extend, extinguish.
None of these require compilers, but you're usually better off for them, especially for serialization and deserialization, regardless of whether you're using a specialized library. On Unix there are libraries that can be used in an ad hoc fashion for RPC/XDR, but also rpcgen (`man rpcgen`).
I haven't done any gRPC programming, but from what I understand it's a modern re-imagining of XML Web Services: you describe the contract and can generate client and server code based on it for different programming languages. Early on in my career I worked on several projects that used WS-* "stack", and generally it was a very good experience. Once we had a project that was split between two subcontractor teams and…
It uses XDR as the schema definition language. XDR is basically analogous to Protobuf files. It has structs and tagged unions and so on.
Another technology from around the same time was DCE/RPC [2]. Microsoft adapted wholesale as MSRPC. Windows used it extensively around the time of NT 3.x for protocols like Exchange Server, and I believe it's still in wide use. DCE/RPC has its own IDL. You used the compiler to generate the stub implementations, just like Protobuf/gRPC.
Microsoft COM uses DCE/RPC under the hood, with lots of extensions [3]. CORBA emerged around the same time as DCE/RPC and COM and is roughly analogous in functionality.
COM and CORBA are explicitly object-oriented. While protocols like DCE/RPC and gRPC return values that are pure data, such as primitives and structs, COM and CORBA can return interfaces. An interface pretends to be a local in-memory instance, but its methods are "stubs" that invoke the underlying RPC call to execute them remotely. Methods can also return functions, which means you have whole trees of objects which are remote. Adding to that, both COM and CORBA use reference counting to hold onto objects, so if a client has received a remote object and reference counted it, the server needs to keep it around until the client either releases the refcount, or the client dies. COM and CORBA called this referential transparency, in that any object could be either local or remote, and a consumer of the interface didn't need to know about it. Of course, while nicely magical, this leads to a lot of complexity. I developed a rather complex distributed DCOM application in the late 1990s, and while it did, inexplicably, work quite well, it was also a nightmare to debug and keep stable.
While COM is alive and well in Windows these days (and interestingly enough, some APIs like DirectX use the COM pattern of defining interfaces via IUnknown etc., but are not actually true COM), DCOM turned out to be a mistake, and CORBA failed for some of the same reasons, although for many reasons unique to CORBA as well. CORBA made tons of design mistakes.
SOAP and WS-*, and of course XML-RPC and JSON-RPC, came later. The wheel has been reinvented many times.
[1] https://datatracker.ietf.org/wg/oncrpc/about/
[2] https://en.wikipedia.org/wiki/DCE/RPC
[3] https://learn.microsoft.com/en-us/openspecs/windows_protocol...