Protobuffers Are Wrong (2018) • Hacker News Board

Re: Protobuffers Are Wrong (2018)

#221

Post by krullin » Sat, Sep 06, 2025, 3:29 PM UTC

kentonv wrote: post #68

Previous discussions: * https://news.ycombinator.com/item?id=18188519 * https://hn.algolia.com/?q=%22Protobuffers+Are+Wrong%22 I guess I'll, once again, copy/paste the comment I made when this was first posted: https://news.ycombinator.com/item?id=18190005 -------- Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto"…

To me, it seems that version-change-safety and the usefulness of the generated code constitute a design tradeoff: If you mark a field as required, then the generated data structures can skip using Option/pointers, and this very common form of validation can be generated for free. If you disallow marking a field as required, then all fields must be checked for existence, even ones required for a system to function, which is quite a burden and will lead to developers having to write their own types anyway as a place to put their validated data into. If data is required to be present for an app to function, then why can't I be given the tools to express this, and benefit from the constraints applied to the data model?

Most of the time when I would like to use a schema-driven, efficient data format and code generation tool, the data contract doesn't change frequently. And when it does, assuming it's a backwards-incompatible change, I think I would be happy to generate a MyDataV2 message along with GetMyDataV2 method, allow existing clients to keep using the original version, and allow new or existing clients to use the newly supported structures at their leisure. Meanwhile, everyone that shares my schema can have much more idiomatic generated code, and in the most common cases won't have to write their own data types or be stuck with a bunch of `if data.x != null {` statements.

Protobufs are an amazing tool, but I think there is a need for a simpler tool which supports a restricted set of use cases cleanly and allows for wider expression of data models.

Re: Protobuffers Are Wrong (2018)

#222

Post by sevensor » Sat, Sep 06, 2025, 3:31 PM UTC

jitl wrote: post #19

lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

Not widely used but I like Typical's approach https://github.com/stepchowfun/typical > Typical offers a new solution ("asymmetric" fields) to the classic problem of how to safely add or remove fields in record types without breaking compatibility. The concept of asymmetric fields also solves the dual problem of how to preserve compatibility when adding or removing cases in sum types.

Seems like a lot of effort to avoid adding a message version field. I’m not a web guy, so maybe I’m missing the point here, but I always embed a schema version field in my data.

Re: Protobuffers Are Wrong (2018)

#223

Post by usrnm » Sat, Sep 06, 2025, 3:43 PM UTC

mkoubaa wrote: post #196

usrnm wrote: post #186

Earlier quoted context omitted.

The thing is a huge pain to manage as a dependency, especially if you wander away from the official google-approved way of doing things. Protobuf went from a breeze to use to the single most common source of build issues in our cross-platform project the moment they added this dependency. It's so bad that many distros and package managers keep the pre-abseil version as a separate package, and many just prefer to get…

I'd rather they just used the abseil headers they needed with the abseil license at the top than make it a build dependency. The concept of a package is antithetical to C++ and no amount of tooling can fix that.

abseil is not header-only, though

Re: Protobuffers Are Wrong (2018)

#224

Post by sdenton4 » Sat, Sep 06, 2025, 3:47 PM UTC

rkagerer wrote: post #175

jitl wrote: post #19

Earlier quoted context omitted.

Not widely used but I like Typical's approach https://github.com/stepchowfun/typical > Typical offers a new solution ("asymmetric" fields) to the classic problem of how to safely add or remove fields in record types without breaking compatibility. The concept of asymmetric fields also solves the dual problem of how to preserve compatibility when adding or removing cases in sum types.

More direct link to the juicy bit: https://github.com/stepchowfun/typical?tab=readme-ov-file#as... An asymmetric field in a struct is considered required for the writer, but optional for the reader.

That's a nice idea... But I believe the design direction of proto buffers was to make everything `optional`, because `required` tends to bite you later when you realize it should actually be optional.

Re: Protobuffers Are Wrong (2018)

#225

Post by 05 » Sat, Sep 06, 2025, 3:50 PM UTC

xmddmx wrote: post #15

I share the author's sentiment. I hate these things. True story: trying to reverse engineer macOS Photos.app sqlite database format to extract human-readable location data from an image. I eventually figured it out, but it was: A base64 encoded Binary Plist format with one field containing a ProtoBuffer which contained another protobuffer which contained a unicode string which contained improperly encoded data (for e…

It that's any consolation, in the current version's schema they are just plain ZLATITUDE FLOAT, ZLONGITUDE FLOAT in ZASSET table..

Re: Protobuffers Are Wrong (2018)

#226

Post by compiler-guy » Sat, Sep 06, 2025, 4:02 PM UTC

tshaddox wrote: post #101

jilles wrote: post #8

Earlier quoted context omitted.

The best way to get your point across is by starting with ad-hominem attacks to assert your superior intelligence.

IMO it's a pretty reasonable claim about experience level, not intelligence, and isn't at all an ad hominem attack because it's referring directly to the fundamental design choices of protocol buffers and thus is not at all a fallacy of irrelevance.

Whatever else Jeff Dean and Sanjay Ghemawat are, and whatever mistakes they made in designing protobufs, they are not amateurs.

Not long after they designed and implemented protobuffers, they shared the ACM prize in computing, as well as many other similar honors. And the honors keep stacking up.

None of this means that protobufs are perfect (or even good), but it does mean they weren't amateurs when they did it.

https://en.wikipedia.org/wiki/Jeff_Dean

https://en.wikipedia.org/wiki/Sanjay_Ghemawat

Re: Protobuffers Are Wrong (2018)

#227

Post by MountainTheme12 » Sat, Sep 06, 2025, 4:07 PM UTC

giveita wrote: post #170

MountainTheme12 wrote: post #35

I agree with the author that protobuf is bad and I ran into many of the issues mentioned. It's pretty much mandatory to add version fields to do backwards compatibility properly. Recently, however, I had the displeasure of working with FlatBuffers. It's worse.

Out of interest why not make the version part of say the URL?

That one was used to implement save data in a game.

Re: Protobuffers Are Wrong (2018)

#228

Post by sdenton4 » Sat, Sep 06, 2025, 4:09 PM UTC

gettingoverit wrote: post #206

senorrib wrote: post #201

Earlier quoted context omitted.

It works both ways. General constructs tend to become overly abstract and you end up with sneaky errors in different places due to a minor change to an abstraction. Like the old adage, this is just a matter of preference. Good software engineering requires, first and foremost, great discipline, regardless of the path or tool you choose.

If there are errors in implementation of general constructs, they tend to be visible at their every use, and get rapidly fixed. Some general constructs are better than the others, because they have an algebraic theory behind them, and sometimes that theory was already researched for a few hundred years. For example, product/coproduct types mentioned in the article are quite close to addition and multiplication that w…

Hm, that's idealistic...

I've certainly run into cases where small changes in general systems led to hard-to-detect bugs, which took a great deal of investigation to figure out. Not all failures are catastrophic.

The technology is quite alive, which is why it hasn't been 'fixed' - changing the wheels on a moving car, and all that. The actual disappointment is that a better alternative hasn't taken off in the six years since this post was written... If its so easy, where's the alternatives?

Re: Protobuffers Are Wrong (2018)

#229

Post by TZubiri » Sat, Sep 06, 2025, 4:10 PM UTC

ndr wrote: post #3

Not even before the first line ends you get "They’re clearly written by amateurs". This is a rage bait, not worth the read.

> if (m_foo = null)

Imagine calling google amateurs, and then the only code you write has a first year student error in failing to distinguish assignment from comparision operator.

There's a class of rant on the internet where programmers complain about increasingly foundational tech instead of admitting skill issues. If you go far deep into that hole, you end up rewriting the kernel in Rust.

Re: Protobuffers Are Wrong (2018)

#230

Post by lostdog » Sat, Sep 06, 2025, 4:15 PM UTC

kentonv wrote: post #211

instig007 wrote: post #189

Earlier quoted context omitted.

> Let's say you want to add a new required field, if your system receives messages from clients some clients may be sending you old data without the field and now the parse step fails because it detects a missing field. A parser has to (inherently) neither fail (compatibility mode) nor lose the new field (a passthrough mode), nor allow diverging (strict mode). The fact that capnproto/parser authors don't realize that…

Hi, I'm the apparently-FUD-spreading Cap'n Proto author. Sure! You could certainly imagine extending Protobuf or Cap'n Proto with a way to specify validation that only happens when you explicitly request it. You'd then have separate functions to parse vs. to validate a message, and then you can perform strict validation at the endpoints but skip it in middleware. This is a perfectly valid feature idea which many peop…

It gets super frustrating to have to empty/null check fields everywhere you use them, especially for fields that are effectively required for the message to make sense.

A very common example I see is Vec3 (just x, y, z). In proto2 you should be checking for the presence of x,y,z every time you use them, and when you do that in math equations, the incessant existence checks completely obscure the math. Really, you want to validate the presence of these fields during the parse. But in practice, what I see is either just assuming the fields exist in code and crashing on null, or admitting that protos are too clunky to use, and immediately converting every proto into a mirror internal type. It really feels like there's a major design gap here.

Don't get me started on the moronic design of proto3, where every time you see Vec3(0,0,0) you get to wonder whether it's the right value or mistakenly unset.