Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

71–80 of 321 posts

Re: Protobuffers Are Wrong (2018)

#71

Post by beders »

If you opt for non-human-readable wire-formats it better be because of very important reasons. Something about measuring performance and operational costs.

If you need to exchange data with other systems that you don't control, a simple format like JSON is vastly superior. You are restricted to handing over tree-like structures. That is a good thing as your consumers will have no problems reading tree-like structures.

It also makes it very simple for each consumer/producer to coerce this data into structs or objects as they please and that make sense to their usage of the data.

You have to validate the data anyhow (you do validate data received by the outside world, do you?), so throwing in coercing is honestly the smallest of your problems.

You only need to touch your data coercion if someone decides to send you data in a different shape. For tree-like structures it is simple to add new things and stay backwards compatible.

Adding a spec on top of your data shapes that can potentially help consumers generate client code is a cherry on top of it and an orthogonal concern.

Making as little assumptions as possible how your consumers deal with your data is a Good Thing(tm) that enabled such useful(still?) things as the WWW.

Re: Protobuffers Are Wrong (2018)

#72

Post by recursive »

mountainriver wrote: post #11

> Protobuffers correspond to the data you want to send over the wire, which is often related but not identical to the actual data the application would like to work with This sums up a lot of the issues I’ve seen with protobuf as well. It’s not an expressive enough language to be the core data model, yet people use it that way. In general, if you don’t have extreme network needs, then protobuf seems to cause more har…

Protobuf is independent from REST. You can have either one. Or both. Or neither. One has nothing to do with the other.

Re: Protobuffers Are Wrong (2018)

#74

Post by dweis »

Analemma_ wrote: post #9

The "no enums as map keys" thing enrages me constantly. Every protobuf project I've ever worked with either has stringly-typed maps all over the place because of this, or has to write its own function to parse Map into Map from the enums and then remember to call that right after deserialization, completely defeating the purpose of autogenerated types and deserializers. Why does Google put up with this? Surely it's t…

I believe that the reason for this limitation is that not all languages can represent open enums cleanly to gracefully handle unknown enums upon schema skew.

Re: Protobuffers Are Wrong (2018)

#75

Post by ozgrakkurt »

iamdelirium wrote: post #58
thinkharderdev wrote: post #48

Earlier quoted context omitted.

Yeah, as soon as you have a moderately complex type the generated code is basically useless. Honestly, ~80% of my gripes about protocol buffers could be alleviated by just allowing me to mark a message field as required.

You think you do but you really don't. What happens if you mark a field as required and then you need to delete it in the future? You can't because if someone stored that proto somewhere and is no longer seeing the field, you just broke their code.

Maybe you don’t delete it then?

Re: Protobuffers Are Wrong (2018)

#76

Post by thinkharderdev »

ants_everywhere wrote: post #30

> Maintain a separate type that describes the data you actually want, and ensure that the two evolve simultaneously. I don't actually want to do this, because then you have N + 1 implementations of each data type, where N = number of programming languages touching the data, and + 1 for the proto implementation. What I personally want to do is use a language-agnostic IDL to describe the types that my programs use. Wit…

> I don't actually want to do this, because then you have N + 1 implementations of each data type, where N = number of programming languages touching the data, and + 1 for the proto implementation.

I think this is exactly what you end up with using protobuf. You have an IDL that describes the interface types but then protoc generates language-specific types that are horrible so you end up converting the generated types to some internal type that is easier to use.

Ideally if you have an IDL that is more expressive then the code generator can create more "natural" data structures in the target language. I haven't used it a ton, but when I have used thrift the generated code has been 100x better than what protoc generates. I've been able to actually model my domain in the thrift IDL and end up with types that look like what I would have written by hand so I don't need to create a parallel set of types as a separate domain model.

Re: Protobuffers Are Wrong (2018)

#77

Post by dinobones »

lols, the weird protobuf initialization semantics has caused so many OMGs. Even on my team it lead to various hard to debug bugs.

It's a lesson most people learns the hard way after using PBs for a few months.

Re: Protobuffers Are Wrong (2018)

#79

Post by zigzag312 »

cornstalks wrote: post #54
jitl wrote: post #19

Earlier quoted context omitted.

Not widely used but I like Typical's approach https://github.com/stepchowfun/typical > Typical offers a new solution ("asymmetric" fields) to the classic problem of how to safely add or remove fields in record types without breaking compatibility. The concept of asymmetric fields also solves the dual problem of how to preserve compatibility when adding or removing cases in sum types.

I've never heard of Typical but the fact they didn't repeat protobuf's sin regarding varint encoding (or use leb128 encoding...) makes me very interested! Thank you for sharing, I'm going to have to give it a spin.

It looks similar to how vint64 lib encodes varints. Total length of varint can be determined via the first byte alone.

Re: Protobuffers Are Wrong (2018)

#80

Post by fsmv »

I actually really strongly prefer 0 being identical to unset. If you have an unset state then you have to check if the field is unset every time you use it. Using 0 allows you to make all of your code "just work" when you pass 0 to it so you don't need to check at all.

It's like how in go most structs don't have a constructor, they just use the 0 value.

Also oneof is made that way so that it is backwards compatible to add a new field and make it a oneof with an existing field. Not everything needs to be pure functional programming.

Post reply on HN