31–40 of 230 posts
For example if I want to import some common set of structs into my protos, there isn’t a standardized or wide spread way to do this. Historically I have had to resort to either copying the structs over or importing multiple protoc generated modules in my code (not in my protos).
If there was a ‘go get’ or ‘pip install’ equivalent for protos, that would be immensely useful; for me and my colleagues at least.
A lot of tooling badness comes out of the fact that gRPC integration in its lingua franca, Go, requires manual wiring of protoc. I don't know why or how there isn't a one-liner option there, because my experience with using gRPC in C# has been vastly better: dotnet add package Grpc.Tools // note below and you have the client and server boilerplate (client - give it url and it's ready for use, server - inherit from ba…
Similar experiences with web services via WCF. It was in dealing with anything published that wasn't .Net where it got difficult. PHP services were not complaint with their own WSDL, similar for internal types in Java from some systems. It was often a mess compared to the C# experience, hence everyone moving towards REST or simpler documentation that was easy to one-off as needed, or use an API client.
Protobuf was designed first and foremost for C++. This makes sense. All of Google's core services are in C++. Yes there's Java (and now Go and to some extent Python). I know. But protobuf was and is a C++-first framework. It's why you have features like arena allocation [1].
Internally there was protobuf v1. I don't know a lot about this because it was mostly gone by the time I started at Google. protobuf v2 was (and, I imagine, still is) the dominant form of.
Now, this isn't to be confused with the API version, which is a completely different thing. You would specify this in BUILD files and it was a complete nightmare because it largely wasn't interchangeable. The big difference is with java_api_version = 1 or 2. Java API v1 was built like the java.util.Date class. Mutable objects with setters and getters. v2 changed this to the builder pattern.
At the time (this may have changed) you couldn't build the artifacts for both API versions and you'd often want to reuse key protobuf definitions that other people owned so you ended up having to use v1 API because some deep protobuf hadn't been migrated (and probably never would be). It got worse because sometimes you'd have one dependency on v1 and another on v2 so you ended up just using bytes fields because that's all you could do. This part was a total mess.
What you know as gRPC was really protobuf v3 and it was designed largely for Cloud (IIRC). It's been some years so again, this may have changed, but there was never any intent to migrate protobuf v2 to v3. There was no clear path to do that. So any protobuf v3 usage in Google was really just for external use.
I explain this because gRPC fails the dogfood test. It's lacking things because Google internally doesn't use it.
So why was this done? I don't know the specifics but I believe it came down to licensing. While protobuf v2 was open sourced the RPC component (internally called "Stubby") never was. I believe it was a licensing issue with some dependency but it's been awhile and honestly I never looked into the specifics. I just remember hearing that it couldn't be done.
So when you read about things like poor JSON support (per this article), it starts to make sense. Google doesn't internally use JSON as a transport format. Protobuf is, first and foremost, a wire format for C++-cetnric APIs (in Stubby). Yes, it was used in storage too (eg Bigtable).
Protobuf in Javascriipt was a particularly horrendous Frankenstein. Obviously Javascript doesn't support binary formats like protobuf. You have to use JSON. And the JSON bridges to protobuf were all uniquely awful for different reasons. My "favorite" was pblite, which used a JSON array indexed by the protobuf tag number. With large protobufs with a lot of optional fields you ended up with messages like:
[null,null,null,null,...(700 more nulls)...,null,{/*my message*/}]
Yes, all of it. Google claims gRPC with protobuf yields a 10-11x performance improvement over HTTP. I am skeptical of those numbers because really it comes down to the frequency of data parsing into and out of the protobuf format. At any rate just use JSON with WebSockets. Its stupid simple and still 7-8x faster than HTTP with far less administrative overhead than either HTTP or gRPC.
That... doesn't make any sense, since gRPC is layered on top of HTTP. There must be missing context here.
Its pretty ironic but Microsoft decided to lean into gRPC support for C#/ASP.NET and its honestly really well done and has great devx.
Why is this ironic?
Yes, all of it. Google claims gRPC with protobuf yields a 10-11x performance improvement over HTTP. I am skeptical of those numbers because really it comes down to the frequency of data parsing into and out of the protobuf format. At any rate just use JSON with WebSockets. Its stupid simple and still 7-8x faster than HTTP with far less administrative overhead than either HTTP or gRPC.
> At any rate just use JSON with WebSockets. Its stupid simple and still 7-8x faster than HTTP with far less administrative overhead than either HTTP or gRPC. gRPC is not supposed to be a standard web communication layer. There are times where you need a binary format and extremely fast serialization/deserialization. Video games are one example where binary formats are greatly preferred over JSON. But I do agree that…
There's almost no reason why RPC should not just be
send(sk, (void *)&mystruct, sizeof(struct mystructtype), 0)A lot of tooling badness comes out of the fact that gRPC integration in its lingua franca, Go, requires manual wiring of protoc. I don't know why or how there isn't a one-liner option there, because my experience with using gRPC in C# has been vastly better: dotnet add package Grpc.Tools // note below and you have the client and server boilerplate (client - give it url and it's ready for use, server - inherit from ba…
The GoGoProtobuf [1] project was started to improve both. It would generate nice Go types that followed Go's conventions. And it uses fast binary serialization without needing to resort to reflection.
Unfortunately, the gRPC/Protobuf team(s) at Google is famously resistant to changes, and was unwilling to work with the GoGo. As a result, the GoGo project is now dead. [2]
I've never used Buf, but it looks like it might fix most of the issues with the Go support.
Something I didn’t see listed was the lack of a package manager for protos. For example if I want to import some common set of structs into my protos, there isn’t a standardized or wide spread way to do this. Historically I have had to resort to either copying the structs over or importing multiple protoc generated modules in my code (not in my protos). If there was a ‘go get’ or ‘pip install’ equivalent for protos,…
Yes, all of it. Google claims gRPC with protobuf yields a 10-11x performance improvement over HTTP. I am skeptical of those numbers because really it comes down to the frequency of data parsing into and out of the protobuf format. At any rate just use JSON with WebSockets. Its stupid simple and still 7-8x faster than HTTP with far less administrative overhead than either HTTP or gRPC.
> because really it comes down to the frequency of data parsing into and out of the protobuf format. Protobuf is intentionally designed to NOT require any parsing at all. Data is serialized over the wire (or stored on disk) in the same format/byte order that it is stored in memory (Yes, that also means that it's not validated at runtime) Or are you referencing the code we all invariably write before/after protobuf to…
As others have mentioned, this is simply not the case, and the VARINT encoding is a trivial counterexample.
It is this required decoding/parsing that (largely) distinguishes protobuf from Google's flatbuffers:
https://github.com/google/flatbuffers
Cap'n Proto (developed by Kenton Varda, the former Google engineer who, while at Google, re-wrote/refactored Google's protobuf to later open source it as the library we all know today) is another example of zero-copy (de)serialization.