Arguing against using protobuffers • Hacker News Board

Re: Arguing against using protobuffers

#61

Post by malkia » Wed, Oct 10, 2018, 10:20 PM UTC

I spent 2.5 years at Google, and most of what I did was pushing one protobuf from one place to another :) - and I loved it... Honestly though, you can complain all day, but some of the decisions made in the list you presented most likely come from experience (daily) not as a user of protobufs (which I simply was), but someone that had to support a plethora of compression formats, how protobufs gets stored in the different databases, proxying, etc. etc.

Your "oneoff" and "repeated" might've been (I dunno really), a decision based on metrics where encoding repeated oneoff would've cost more, or made problems, more testing, or who knows what.

For one, Google was, and I believe still is open to discussing all kinds of matters, and you can get on to a design doc and comment if you like, if you care, if you have something to say... You can definitely influence, and people really did talk (you can even experience this from their public design docs - like from chromium, etc.).

And I love protobufs, so I guess I'm biased...

Re: Arguing against using protobuffers

#62

Post by ericpauley » Wed, Oct 10, 2018, 10:20 PM UTC

tathougies wrote: post #48

Easiest way to send data over the wire. struct mytype s; s.field1 = something; s.field2 = something else; send(socket, &s, sizeof(s), 0) or, using a language with a good type system like Haskell data MyData = MyData Int Int deriving Generic instance Storable MyData alloca $ \buf -> do let d = MyData field1 field2 poke buf d send socket buf (sizeof d)

C doesn't make struct memory layout guarantees, endian guarantees, or size guarantees in some cases, so unless you're running the exact same binary this is a very poor serialization technique.

Re: Arguing against using protobuffers

#63

Post by amluto » Wed, Oct 10, 2018, 10:21 PM UTC

tathougies wrote: post #48

Easiest way to send data over the wire. struct mytype s; s.field1 = something; s.field2 = something else; send(socket, &s, sizeof(s), 0) or, using a language with a good type system like Haskell data MyData = MyData Int Int deriving Generic instance Storable MyData alloca $ \buf -> do let d = MyData field1 field2 poke buf d send socket buf (sizeof d)

This may be easy, but it’s wrong. Endianness issues are just the start. Information leaks due to padding are a big deal. And it straight up doesn’t work if nontrivial data structures are involved.

Re: Arguing against using protobuffers

#64

Post by romed » Wed, Oct 10, 2018, 10:21 PM UTC

_fq4v wrote: post #46

romed wrote: post #33

It's abundantly clear why this author lasted only a year at Google. Aside from using the non-idiomatic "protobuffers" ... "nothing wants to inspect only some bits of a message and then forward it on unchanged" In my experience it is extremely useful to partially parse a protobuf, or to not parse it at all and simply modify it by appending to it. Also useful is the ability to define a message type that is isomorphic o…

Ah right... when you can't attack the ideas, attack the author. Sandy is a Haskell enthusiast, and it is not a surprise that his criticism of protobuf is inspired heavily by that. The argument presented is that protobuf does not attempt to even replicate the best practices we already have in terms of data representation. It makes sense why languages like C are constrained in their data representations -- they want to…

Sure that makes sense if you can write off C++ as a "legacy" language, which is fine if you just can't get your mind wrapped around the scale at which Google operates. Due to the nature of weighted averages, you can't just write off something as being a "Google-only problem". Google and its peers like Amazon and Facebook own a very large fraction of the world's computing resources.

I think the overall mistake you and the author are making is assuming it is desirable for one program examining an encoded message to respect the type system of the program that produced it. That assumption is not obviously correct. The flexibility to just treat a vector of numbers as a vector of numbers, or to skip it, or ignore it, etc are pretty important in practice. That is why protobuf is defined at the level where it is defined. It encodes numbers and strings on the wire. It happens to be a fairly good way to represent search data. It brings no CS academic wankery to the party. It's very practical.

Re: Arguing against using protobuffers

#65

Post by enneff » Wed, Oct 10, 2018, 10:21 PM UTC

atombender wrote: post #25

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code gen…

An approach I've used on the Go side is to build nice native Go types and converters to/from proto representations for wire marshaling. There's a minor performance hit but it's insignificant for what I'm doing. The result is quite nice, and you can largely ignore protobufs except for defining service APIs (which it's actually decent at).

Which tool are you using for TypeScript protobuf code generation? This is the next step for me, and I'd be keen to hear more of your experience there.

Re: Arguing against using protobuffers

#66

Post by dweis » Wed, Oct 10, 2018, 10:22 PM UTC

docker_up wrote: post #40

Put me firmly in the camp of "optional fields are bad." I believe all fields should be required. I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility. The idea of "forwards compatibility" is a pipedream, in my opinion.…

How do you handle binary rollbacks and rollouts safely with an "everything is required" approach? Do you force binaries to roll out in a strict order with appropriate soak time at each layer? How does that affect developer velocity?

Re: Arguing against using protobuffers

#67

Post by hardwaresofton » Wed, Oct 10, 2018, 10:23 PM UTC

atombender wrote: post #25

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code gen…

I'd like to humbly suggest that we use JSON please, in particular:

JSON + JSONSchema[0] +/- JSON Hyperschema[1] +/- JSON LD[2]

It's a bit to learn but I promise you, it's worth it. The technologies are not redundant (jsonschema spec is for validation, hyperschema spec is for specifying how you interact, and LD is for semantics like language and more). If you take a few hours, read all 3 specs, you're almost guaranteed to imagine a new (albeit somewhat daunting) future, where APIs and clients can be smarter.

There's no reason I can think of that a sufficiently specified JSON schema cannot be automatically converted into a efficient binary format. I have absolutely zero reason to use Protobuf and it's whole ecosystem of tools if that's the case, I'd much rather go with tools built on a standard like JSON.

GraphQL's functionality is a subset of the above set of tools -- IIRC GraphQL can basically be boiled down to an option (or two) on a resource endpoint -- How you query on the server side does indeed change a bit to support the syntax, but this is basically the same as if you'd implemented with horizontal filtering like in POSTgrest[3], or resource embedding feature[4] (which everyone ends up doing at some point with REST-ish APIs). I honestly don't know who or what is trying so hard to drive GraphQL adoption so hard but I think it's mostly hype, and in terms of general technology, by default it forces clients to know the inner workings of the data model and that's bad for any kind of future in which you don't want to have to learn X APIs to interact with X systems.

BTW I absolutely love a good well-supported rant that agrees with all my biases (yes, Google as well as every other company has a bunch of amateurs, as well as a small amount of very skilled engineers), and all the problems noted with protobuf are real problems, and this viewpoint is sorely missed in all the talks and conferences where people are touting GRPC as the next sliced bread with Protobufs as the transport (you can change GRPC's transport layer to be JSON for example[5])

[0]: https://json-schema.org/

[1]: https://datatracker.ietf.org/doc/draft-handrews-json-schema-... (also on json-schema.org)

[2]: https://json-ld.org/

[3]: https://postgrest.org/en/v5.1/api.html#horizontal-filtering-...

[4]: https://postgrest.org/en/v5.1/api.html#resource-embedding

[5]: https://grpc.io/blog/grpc-with-json

Re: Arguing against using protobuffers

#68

Post by chabsf » Wed, Oct 10, 2018, 10:23 PM UTC

tathougies wrote: post #56

chabsf wrote: post #41

Earlier quoted context omitted.

Furthermore, in the case where I'm using protobufs as a necessary evil of grpc, what's the best alternative here?

JSON/custom binary format + HTTP2?

HTTP2 is just the transport, there would be a ton of machinery to implement in each language in the stack.

Re: Arguing against using protobuffers

#69

Post by enneff » Wed, Oct 10, 2018, 10:23 PM UTC

weberc2 wrote: post #16

CobrastanJorji wrote: post #6

Earlier quoted context omitted.

I've heard good things about https://capnproto.org/ but the author of this piece may strongly disagree, since I think their objections are rather more fundamental, and also because Cap'n Proto went with banning the concept of "required" fields. I'd also be interested in what alternatives the author might suggest.

Capnproto is built by the guy who originally built protobufs, and he cites capnproto as being better because he was able to learn from his mistakes. Makes me optimistic.

Minor clarification: the author of Capnproto worked on protobufs at Google but was not their original creator.

Re: Arguing against using protobuffers

#70

Post by mlthoughts2018 » Wed, Oct 10, 2018, 10:23 PM UTC

mixmastamyk wrote: post #39

sonnyblarney wrote: post #2

Good points, but can someone articulate what the best alternative to protobuffers would be in 2018, you know with 'hindsight' etc.?

Apache Thrift is another one.

Thrift suffers pretty much all the same problems as protobufs and has many similar dysfunctional failure modes, like places that rely on serializing into Thrift structs stores in hdfs and treating that like a de facto database, with Thrift struct definitions as the schema. It is so miserable to work in code bases like that.