Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

111–120 of 321 posts

Re: Protobuffers Are Wrong (2018)

#111

Post by thethimble »

summerlight wrote: post #94

https://news.ycombinator.com/item?id=18190005 Just FYI: an obligatory comment from the protobuf v2 designer. Yeah, protobuf has lots of design mistakes but this article is written by someone who does not understand the problem space. Most of the complexity of serialization comes from implementation compatibility between different timepoints. This significantly limits design space.

Relatedly, most of the author's concerns are solved by wrapping things in a message.

> oneof fields can’t be repeated.

Wrap oneof field in message which can be repeated

> map fields cannot be repeated.

Wrap in message which can contain repeated fields

> map values cannot be other maps.

Wrap map in message which can be a value

Perhaps this is slightly inconvenient/un-ergonomic, but the author is positioning these things as "protos fundamentally can't do this".

Re: Protobuffers Are Wrong (2018)

#112

Post by BinaryIgor »

Avro (and others) has its own set of problems as well.

For messaging, JSON, used in the same way and with the same versioning practices as we have established for evolving schemas in REST APIs, has never failed me.

It seems to me that all these rigid type systems for remote procedure calls introduce more problems that they really solve and bring unnecessary complexity.

Sure, there are tradeoffs with flexible JSONs - but simplicity of it beats the potential advantages we get from systems like Avro or ProtoBuf.

Re: Protobuffers Are Wrong (2018)

#113

Post by jcgrillo »

lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

As someone who has written many mapreduce jobs over years old protobufs I can confidently report the backwards compatibility made it possible at all.

Re: Protobuffers Are Wrong (2018)

#114

Post by btilly »

ndr wrote: post #3

Not even before the first line ends you get "They’re clearly written by amateurs". This is a rage bait, not worth the read.

The reasons for that line get at a fundamental tension. As David Wheeler famously said, "All problems in computer science can be solved by another level of indirection, except for the problem of too many indirections."

Over time we accumulate cleverer and cleverer abstractions. And any abstraction that we've internalized, we stop seeing. It just becomes how we want to do things, and we have no sense of what cost we are imposing with others. Because all abstractions leak. And all abstractions pose a barrier for the maintenance programmer.

All of which leads to the problem that Brian Kernighan warned about with, "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?" Except that the person who will have to debug it is probably a maintenance programmer who doesn't know your abstractions.

One of the key pieces of wisdom that show through Google's approaches is that our industry's tendency towards abstraction is toxic. As much as any particular abstraction is powerful, allowing too many becomes its own problem. This is why, for example, Go was designed to strongly discourage over-abstraction.

Protobufs do exactly what it says on the tin. As long as you are using them in the straightforward way which they are intended for, they work great. All of his complaints boil down to, "I tried to do some meta-manipulation to generate new abstractions, and the design said I couldn't."

That isn't the result of them being written by amateurs. That's the result of them being written to incorporate a piece of engineering wisdom that most programmers think that they are smart enough to ignore. (My past self was definitely one of those programmers.)

Can the technology be abused? Do people do stupid things with them? Are there things that you might want to do that you can't? Absolutely. But if you KISS, they work great. And the more you keep it simple, the better they work. I consider that an incentive towards creating better engineered designs.

Re: Protobuffers Are Wrong (2018)

#115

Post by stickfigure »

lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

Backwards compatibility is just not an issue in self-describing structures like JSON, Java serialization, and (dating myself) Hessian. You can add fields and you can remove fields. That's enough to allow seamless migrations.

It's only positional protocols that have this problem.

Re: Protobuffers Are Wrong (2018)

#116

Post by sieabahlpark »

recursive wrote: post #72
mountainriver wrote: post #11

> Protobuffers correspond to the data you want to send over the wire, which is often related but not identical to the actual data the application would like to work with This sums up a lot of the issues I’ve seen with protobuf as well. It’s not an expressive enough language to be the core data model, yet people use it that way. In general, if you don’t have extreme network needs, then protobuf seems to cause more har…

Protobuf is independent from REST. You can have either one. Or both. Or neither. One has nothing to do with the other.

[dead]

Re: Protobuffers Are Wrong (2018)

#117

Post by jandrese »

nicce wrote: post #55
mountainriver wrote: post #11

> Protobuffers correspond to the data you want to send over the wire, which is often related but not identical to the actual data the application would like to work with This sums up a lot of the issues I’ve seen with protobuf as well. It’s not an expressive enough language to be the core data model, yet people use it that way. In general, if you don’t have extreme network needs, then protobuf seems to cause more har…

On the other hand, ASN.1 is very expressive and can cover pretty much anything, but Protobuff was created because people thought ASN.1 is too complex. I guess we can't have both.

"Those who cannot remember the past are condemned to repeat it" -- George Santayana

Re: Protobuffers Are Wrong (2018)

#118

Post by jandrese »

taeric wrote: post #2

I'm more than a little curious what event caused such a strong objection to protobuffers. :D I do tend to agree that they are bad. I also agree that people put a little too much credence in "came from Google." I can't bring myself to have this much anger towards it. Had to have been something that sparked this.

As a developer I always see "came from Google" as a yellow flag.

Too often I find something mildly interesting, but then realize that in order for me to try to use it I need to set up a personal mirror of half of Google's tech stack to even get it to start.

Re: Protobuffers Are Wrong (2018)

#119

Post by danans »

ants_everywhere wrote: post #30

> Maintain a separate type that describes the data you actually want, and ensure that the two evolve simultaneously. I don't actually want to do this, because then you have N + 1 implementations of each data type, where N = number of programming languages touching the data, and + 1 for the proto implementation. What I personally want to do is use a language-agnostic IDL to describe the types that my programs use. Wit…

> The practical alternative is to use JSON everywhere, possibly with some additional tooling to generate code from a JSON schema.

Protobuf has a bidirectional JSON mapping that works reasonably well for a lot of use cases.

I have used it to skip the protobuf wire format all together and just use protobuf for the IDL and multi-language binding, both of which IMO are far better than JSON-Schema.

JSON-Schema is definitely more powerful though, letting you do things like field level constraints. I'd love to see you tomorrow that paired the best of both.

Re: Protobuffers Are Wrong (2018)

#120

Post by jimbokun »

stickfigure wrote: post #115
lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

Backwards compatibility is just not an issue in self-describing structures like JSON, Java serialization, and (dating myself) Hessian. You can add fields and you can remove fields. That's enough to allow seamless migrations. It's only positional protocols that have this problem.

At the cost of much larger payloads.
Post reply on HN