Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

191–200 of 321 posts

Re: Protobuffers Are Wrong (2018)

#191

Post by tliltocatl »

henningpeters wrote: post #188
tliltocatl wrote: post #181

Earlier quoted context omitted.

Embedded/constrained UDP is where protobuf wire format (but not google's libraries) rocks: IoT over cellular and such, where you need to fit everything into a single datagram (number of roundtrips is what determines power consumption). As to those who say "UDP is unreliable" - what you do is you implement ARQ on the application level. Just like TCP does it, except you don't have to waste roundtrips on SYN-SYN-ACK han…

> why on earth would you waste precious bytes on schema every time cbor doesn't prescribe sending schema, in fact there is no schema, like json. i just switched from protobuf to cbor because i needed better streaming support and find use it quite delightful. losing protobuf schema hurts a bit, but the amount of boilerplate code is actually less than what i had before with nanopb (embedded context). on top of it, i am…

> cbor doesn't prescribe sending schema, in fact there is no schema, like json.

You are right, I must have confused CBOR with BSON where you send field names as strings.

>on top of it, i am saving approx. 20% in message size compared to protobuf bc i am using mostly arrays with fixed position parameters

Arrays with fixed position is always going to be the most compact format, but that means that you essentially give up on serialization. Also, when you have a large structure (e. g. full set of device state and settings)where most of the fields only change infrequently, it makes sense to only send what's changed, and then TLV is significantly better.

Re: Protobuffers Are Wrong (2018)

#192

Post by instig007 »

nice_byte wrote: post #129
nice_byte wrote: post #123

> Make all fields in a message required. funnily enough, this line alone reveals the author to be an amateur in the problem space they are writing so confidently about.

the complaints about the Protobuf type system being not flexible enough are also really funny to read. fundamentally, the author refuses to contend with the fact that the context in which Protobufs are used -- millions of messages strewn around random databases and files, read and written by software using different versions of libraries -- is NOT the same scenario where you get to design your types once and then EVE…

> is NOT the same scenario where you get to design your types once and then EVERYTHING that ever touches those types is forced through a type checker.

the author never claimed the types had to be designed only once, he claimed that schema evolution chosen by protobuf is inadequate for the purpose of lossless evolution.

> Kenton has already provided a good explanation here: https://news.ycombinator.com/item?id=45140590

TLDR: yada-yada [...] protobuf is practical, type algebra either doesn't exist or impractical because only PL theorists know about it, not Kenton.

Re: Protobuffers Are Wrong (2018)

#193

Post by frumiousirc »

stinkbeetle wrote: post #182

With these serialization libraries, do any of them have a facility that allows you to specify a wire format and an application format, with recipes for converting one to the other? I haven't used these very seriously but a problem I had a while back was that that the wire format was not what the applications wanted to use, but a good application format was to space-inefficient for wire. As far as I could see there wa…

The way to do this starts with not hard-wiring the code generation step.

Instead, make codegen a function of BOTH a data schema object and a code template (eg expressed in Jinja2 template language - or ZeroMQ GSL where I first saw this approach). The codegen stage is then simply the application of the template to the data schema to produce a code artifact.

The templates are written assuming the data schema is provided following a meta-schema (eg JSON Schema for a data schema in JSON). One can develop, eg per-language templates to produce serialization code or intra-language converters between serialization forms (on wire) and application friendly forms. The extra effort to develop a template for a particular target is amortized as it will work across all data schemas that adhere to a common meta-schema.

The "codegen" stage can of course be given non "code" templates to produce, eg, reference documentation about the data schema in different formats like HTML, text, nroff/man, etc.

Re: Protobuffers Are Wrong (2018)

#195

Post by mkoubaa »

lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

Real ones know that serialization is what sucks.

Re: Protobuffers Are Wrong (2018)

#196

Post by mkoubaa »

usrnm wrote: post #186
porridgeraisin wrote: post #177

Earlier quoted context omitted.

I always thought people had a positive view on abseil, never used it myself other than when tinkering on random projects. What's the main issue?

The thing is a huge pain to manage as a dependency, especially if you wander away from the official google-approved way of doing things. Protobuf went from a breeze to use to the single most common source of build issues in our cross-platform project the moment they added this dependency. It's so bad that many distros and package managers keep the pre-abseil version as a separate package, and many just prefer to get…

I'd rather they just used the abseil headers they needed with the abseil license at the top than make it a build dependency.

The concept of a package is antithetical to C++ and no amount of tooling can fix that.

Re: Protobuffers Are Wrong (2018)

#197

Post by tombert »

gethly wrote: post #65

I too was using PBs a lot, as they are quite popular in the Go world. But i came to the conclusion that they and gRPC are more trouble than they are worth. I switched to JSON, HTTP "REST" and websockets, if i need streaming, and am as happy as i could be. I get the api interoperability between various languages when one wants to build a client with strict schema but in reality, this is more of a theory than real life…

I am very partial to msgpack. It has routinely met or exceeded my performance needs and doesn’t depend on weird code generation, and is super easy to set up.

Something that I don’t see talked about much with msgpack, but I think is cool: if your project doesn’t span across multiple languages, you can actually embed those language semantics into your encoder with extensions.

For example, in Clojure’s port of msgpack out of the box, you can use Clojure keywords out of the box and it will parse correctly without issue. You also can have it work with sets.

Obviously you could define some kind of mapping yourself and use any binary format to do this, ultimately the [en|de]coder is just using regular msgpack constructs behind the scenes, but i have always had to do that manually while with msgpack it seems like the libraries readily embrace it.

Re: Protobuffers Are Wrong (2018)

#199

Post by the__alchemist »

lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

This is always the thing to look for; "What are the alternatives?", and/why aren't there better ones.

I don't understand most use cases of protobufs, including ones that informed their design. I use it for ESP-hosted, to communicate between two MCUs. It is the highest-friction serialization protocol I've seen, and is not very byte-efficient.

Maybe something like the specialized serialization libraries (bincode, postcard etc) would be easier? But I suspect I'm missing something about the abstraction that applies to networked systems, beyond serialization.

Re: Protobuffers Are Wrong (2018)

#200

Post by co_dh »

But why do you need serialization? Because the data structure on disk is not the same as in memory. Arthur Whitney's k/q/kdb+ solved this problem by making them the same. An array has the same format in memory and on disk, so there is no serialization, and even better, you can mmap files into memory, so you don't need cache!

He also removed the capability to define a structure, and force you to use dictionary(structure) of array, instead of array of structure.

Post reply on HN