Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

21–30 of 307 posts

Re: Arguing against using protobuffers

#21
post #14
post #11

Mostly lost me at "Make all fields in a message required. This makes messages product types." Your constraints on protocols can change over time and context, changing something that is required to something that is optional can cause crashes in prod. Unless somehow the 'optional' design has a way to handle that? You also will have to do custom validation anyways, as your storage format will never be able to enforce a…

If you would continue reading, there is an explanation of how optional would work almost immediately after this quote.

Right - this is the same as having the possibility of both required and optional. I'm saying there should not be any possibility of a 'required' field

Re: Arguing against using protobuffers

#22

Good points, but can someone articulate what the best alternative to protobuffers would be in 2018, you know with 'hindsight' etc.?

JSON or graphql. Personally I think the typing, API compatibility and extensive language support make protobuffers currently the winner though, despite the warts

How would graphql help reduce the latency of API calls between services?

Re: Arguing against using protobuffers

#23
post #10

Versioning was the biggest disappointment for me. I just want a middleware layer that can handle clients of different versions reliably. Surely everyone has the same problem.

>Surely everyone has the same problem.

Yes, we do.

But the appropriate answer in most cases is "it's up to the business logic". Trying to solve it at the transport layer is futile.

Re: Arguing against using protobuffers

#24

Good points, but can someone articulate what the best alternative to protobuffers would be in 2018, you know with 'hindsight' etc.?

JSON or graphql. Personally I think the typing, API compatibility and extensive language support make protobuffers currently the winner though, despite the warts

Neither of those are binary formats though, and if I’m picking up a binary format like Protobuffers it’s probably because I want the performance benefits.

Personally, these days I’m taking the approach of just outright avoiding pretty much anything google does. Either because it solves a problem only they have, or they’ll probably just drop it out of the blue one day. Or like Tensorflow, it’s fucking impossible to get your head around because their documentation seems to be written with the goal of being maximally confusing.

Re: Arguing against using protobuffers

#25
Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration.

The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code generator, but the Go generator is just egregiously bad. There's no way to write a client or server that uses the Protobuf structs directly as first-class data types without ending up looking like a ball of spaghetti, at least not if you venture into the realm of oneofs, timestamps or the "well known types" set of types.

I think we're at a juncture where we really desperately need a better, unified way to express schemas and pass type-safe data over the network that flows through these schemas. I'm currently working on a project that involves both gRPC (which uses Protobuf), GraphQL and JSON Schema, with the backends written in Go, and the frontend in TypeScript, and the overlap between all of these is ridiculous. TypeScript is a fresh breath of air that mostly solves the brittleness of JavaScript's type system, and it's absolutely crucial to extend this type-safety all the way to the backend.

The challenge is that no language is the same, and a common schema format ends up targeting a lowest-common denominator. For example, GraphQL errs (in my opinion, wrongly) on the side of simplicity and doesn't support maps, so now you have a whole layer that needs to either custom types (JSON as an escape hatch, basically) or emulate maps using arrays of pairs, neither of which is ideal.

Re: Arguing against using protobuffers

#26
I will say one thing. gzipped JSON is damn efficient over the wire. It elegantly solves the problem that field names are repeated in every object. The only problem is that adding compression and a serializationd/deserialization step cost CPU time. However if you have this problem then you're often better off by using something like cap'n proto or flatbuffers which do not have an extra serdes step. If you consider this then there is only a very very narrow band in which it makes sense to use protobuffers vs JSON (which is a defacto standard) or lesser known solutions that provide superior performance.

Re: Arguing against using protobuffers

#28
> Option 1 is clearly the "right" solution, but its untenable with protobuffers. The language isn't powerful enough to encode types that can perform double-duty as both wire and application formats. Which means you'd need to write a completely separate datatype, evolve it synchronously with the protobuffer, and explicitly write serialization code between the two. Seeing as most people seem to use protobuffers in order to not write serialization code, this is obviously never going to happen.

This last is not the case. I've only ever seen it used when people a) already have written serialization code that works with XML/JSON, and b) don't want to write code to serialize to a binary format.

JSON also doesn't let you use objects as object keys, or have any form of polymorphism. It seems like the author wants protobufs to be an RPC system with a modern type system, when in fact it is a data interchange format.

Re: Arguing against using protobuffers

#29

Wow, some minor complaints on language formality issues, then discredit the thing as a whole. But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential. It's like complaining a car that drives fast and has great gas efficiency of cannot talk. (Well in 2018, it might be less of a stretch to demand a car that can…

> But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential.

Huh? It's a serialization format. This problem has been solved many times in many different ways.

The author is pointing out that Protobufs were designed poorly, and they didn't have to be.

Post reply on HN