Protobuffers Are Wrong (2018) • Hacker News Board

Re: Protobuffers Are Wrong (2018)

#241

Post by summerlight » Sat, Sep 06, 2025, 5:14 PM UTC

instig007 wrote: post #171

summerlight wrote: post #132

Earlier quoted context omitted.

If you see some statements like below on the serialization topic: > Make all fields in a message required. This makes messages product types. > One possible argument here is that protobuffers will hold onto any information present in a message that they don't understand. In principle this means that it's nondestructive to route a message through an intermediary that doesn't understand this version of its schema. Sure…

> If you see some statements like below on the serialization topic: > Make all fields in a message required. This makes messages product types. > Then it is fair to raise eyebrows on the author's expertise. It's fair to raise eyebrows on your expertise, since required fields don't contribute to b/w incompatibility at all, as every real-world protocol has a mandatory required version number that's tied to a direct par…

> since required fields don't contribute to b/w incompatibility at all, as every real-world protocol has a mandatory required version number that's tied to a direct parsing strategy with strictly defined algebra

At least I know 10 different tech companies with billion dollars revenue which does not suit to your description. This comment makes me wonder if you have any experience of working on real world distributed systems. Oh and I'm pretty sure you did not read Kenton's comment; he already precisely addressed your point:

> This is especially true when it comes to protocols, because in a distributed system, you cannot update both sides of a protocol simultaneously. I have found that type theorists tend to promote "version negotiation" schemes where the two sides agree on one rigid protocol to follow, but this is extremely painful in practice: you end up needing to maintain parallel code paths, leading to ugly and hard-to-test code. Inevitably, developers are pushed towards hacks in order to avoid protocol changes, which makes things worse.

I recommend you to do your homework before making such a strong argument. Reading a 5 mins long comment is not that hard. You can avoid lots of shame by doing so.

Re: Protobuffers Are Wrong (2018)

#242

Post by thinkharderdev » Sat, Sep 06, 2025, 5:15 PM UTC

taeric wrote: post #135

thinkharderdev wrote: post #93

Earlier quoted context omitted.

But this is something you should be able to handle on a case-by-case basis. If you have a type which is stored durably as protobuf then adding required fields is much harder. But if you are just dealing with transient rpc messages then it can be done relatively easily in a two step process. First you add the field as optional and then once all producers are upgraded (and setting the new field), make it required. It's…

Largely true. If you are at Google scale, odds are you have mixed fleets deployed. Such that it is a bit of involved process. But it is well defined and doable. I think a lot of us would rather not do a dance we don't have to do?

Sure, you just have to balance that against the cost of a poorly specified API interface. The errors because clients aren't clear on what is really required or not, what they should consider an error if it is not defined, etc. And of course all the boilerplate code that you have to write to convert the interface model to an internal domain model you can actually use inside your code.

Re: Protobuffers Are Wrong (2018)

#243

Post by mdaniel » Sat, Sep 06, 2025, 5:16 PM UTC

frumiousirc wrote: post #193

stinkbeetle wrote: post #182

With these serialization libraries, do any of them have a facility that allows you to specify a wire format and an application format, with recipes for converting one to the other? I haven't used these very seriously but a problem I had a while back was that that the wire format was not what the applications wanted to use, but a good application format was to space-inefficient for wire. As far as I could see there wa…

The way to do this starts with not hard-wiring the code generation step. Instead, make codegen a function of BOTH a data schema object and a code template (eg expressed in Jinja2 template language - or ZeroMQ GSL where I first saw this approach). The codegen stage is then simply the application of the template to the data schema to produce a code artifact. The templates are written assuming the data schema is provide…

I didn't recognize the GSL citation, so for others:

- https://github.com/zeromq/gsl/blob/v4.1.5/examples/fsm_c.gsl

whew, this readme has everything

- XML in, text out: https://github.com/zeromq/gsl#:~:text=feed%20it%20some%20dat...

- a whole section on software engineering https://github.com/zeromq/gsl#model-oriented-programming

- they support COBOL https://github.com/zeromq/gsl#cobol

- and then a project 11 years old with "we're going to document these functions one day" https://github.com/zeromq/gsl#global-functions

What a journey that was

Re: Protobuffers Are Wrong (2018)

#244

Post by motorest » Sat, Sep 06, 2025, 5:20 PM UTC

const_cast wrote: post #130

mattnewton wrote: post #36

Earlier quoted context omitted.

Exactly, I think of protobuffers like I think of Java or Go - at least they weren’t writing it in C++. Dragging your org away from using poorly specified json is often worth these papercuts IMO.

Protobufs are better but not best. Still, by far, the easiest thing to use and the safest is actual APIs. Like, in your application. Interfaces and stuff. Obviously if your thing HAS to communicate over the network that's one thing, but a lot of applications don't. The distributed system micro service stuff is a choice. Guys, distributed systems are hard. The extremely low API visibility combined with fragile network…

> Protobufs are better but not best.

This sort of comments doesn't add anything to the discussion unless you are able to point out what you believe to be the best. It reads as an unnecessary and unsubstantiated put-down.

Re: Protobuffers Are Wrong (2018)

#245

Post by vouwfietsman » Sat, Sep 06, 2025, 5:22 PM UTC

sevensor wrote: post #222

jitl wrote: post #19

Earlier quoted context omitted.

Not widely used but I like Typical's approach https://github.com/stepchowfun/typical > Typical offers a new solution ("asymmetric" fields) to the classic problem of how to safely add or remove fields in record types without breaking compatibility. The concept of asymmetric fields also solves the dual problem of how to preserve compatibility when adding or removing cases in sum types.

Seems like a lot of effort to avoid adding a message version field. I’m not a web guy, so maybe I’m missing the point here, but I always embed a schema version field in my data.

I get that.

The point is that its hard to prevent asymmetry in message versions if you are working with many communicating systems. Lets say four services inter-communicate with some protocol, it is extremely annoying to impose a deployment order where the producer of a message type is the last to upgrade the message schema, as this causes unnecessary dependencies between the release trains of these services. At the same time, one cannot simply say: "I don't know this message version, I will disregard it" because in live systems this will mean the systems go out of sync, data is lost, stuff breaks, etc.

There's probably more issues I haven't mentioned, but long story short: in live, interconnected systems, it becomes important to have intelligent message versioning, i.e: a version number is not enough.

Re: Protobuffers Are Wrong (2018)

#246

Post by awalsh128 » Sat, Sep 06, 2025, 5:30 PM UTC

ndr wrote: post #3

Not even before the first line ends you get "They’re clearly written by amateurs". This is a rage bait, not worth the read.

Yeah, there is a lot of snark in the article which undermines their argument.

Re: Protobuffers Are Wrong (2018)

#247

Post by [deleted] » Sat, Sep 06, 2025, 5:33 PM UTC

motorest wrote: post #217

lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

> Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. What I dislike the most about blog posts like this is that, although the blogger is very opinionated and critical of many things, the post dates back to 2018, protobuf is still dominant, and apparently during all these years the blogger failed to put together something that they felt was a bett…

[deleted]

Re: Protobuffers Are Wrong (2018)

#248

Post by kentonv » Sat, Sep 06, 2025, 5:40 PM UTC

instig007 wrote: post #207

kentonv wrote: post #68

Previous discussions: * https://news.ycombinator.com/item?id=18188519 * https://hn.algolia.com/?q=%22Protobuffers+Are+Wrong%22 I guess I'll, once again, copy/paste the comment I made when this was first posted: https://news.ycombinator.com/item?id=18190005 -------- Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto"…

> Real-world practice has also shown that quite often, fields that originally seemed to be "required" turn out to be optional over time how often? as practiced by who, and where? > 2. You actually do not want a oneof field to be repeated! > How do you make this change without breaking compatibility? Now you wish that you had defined your array as an array of messages, each containing a oneof, so that you could add a…

> how often? as practiced by who, and where?

This was my experience in Google Search infrastructure circa 2005-2010. This was a system with dozens of teams and hundreds of developers all pushing their data through a common message bus.

It happened all the damned time and caused multiple real outages (from overzealous validation), along with a lot of tech debt involving having to initialize fields with dummy data because they weren't used anymore but still required.

Reports from other large teams at google, e.g. gmail, indicated they had the same problems.

> Nice, "explain to me how you're going to implement a backward-compatible SUM in the spec-parser that doesn't have the notions needed. Ha! You can't! Told you so!"

Sure sure, we could expand the type system to support some way of adding a new tag to every element of the repeated oneof, implement it, teach everyone how that works, etc.

Or we could just tell people to wrap the thing in a `message`. It works fine already, and everyone already understands how to do it. No new cognitive load is created, and no time is wasted chasing theoretical purity that provides no actual real-world benefit.

Re: Protobuffers Are Wrong (2018)

#249

Post by jjmarr » Sat, Sep 06, 2025, 5:41 PM UTC

usrnm wrote: post #186

porridgeraisin wrote: post #177

Earlier quoted context omitted.

I always thought people had a positive view on abseil, never used it myself other than when tinkering on random projects. What's the main issue?

The thing is a huge pain to manage as a dependency, especially if you wander away from the official google-approved way of doing things. Protobuf went from a breeze to use to the single most common source of build issues in our cross-platform project the moment they added this dependency. It's so bad that many distros and package managers keep the pre-abseil version as a separate package, and many just prefer to get…

I like abseil besides the compile times. Not having to specialize my own hash when using maps is nice.

Re: Protobuffers Are Wrong (2018)

#250

Post by vineyardmike » Sat, Sep 06, 2025, 5:42 PM UTC

sevensor wrote: post #222

jitl wrote: post #19

Earlier quoted context omitted.

Not widely used but I like Typical's approach https://github.com/stepchowfun/typical > Typical offers a new solution ("asymmetric" fields) to the classic problem of how to safely add or remove fields in record types without breaking compatibility. The concept of asymmetric fields also solves the dual problem of how to preserve compatibility when adding or removing cases in sum types.

Seems like a lot of effort to avoid adding a message version field. I’m not a web guy, so maybe I’m missing the point here, but I always embed a schema version field in my data.

Idk I generally think “magic numbers” are just extra effort. The main annoyance is adding if statements everywhere on version number instead of checking the data field you need being present.

It also really depends on the scope of the issue. Protos really excel at “rolling” updates and continuous changes instead of fixed APIs. For example, MicroserviceA calls MicroserviceB, but the teams do deployments different times of the week. Constant rolling of the version number for each change is annoying vs just checking for the new feature. Especially if you could have several active versions at a time.

It also frees you from actually propagating a single version number everywhere. If you own a bunch of API endpoints, you either need to put the version in the URL, which impacts every endpoint at once, or you need to put it in the request/response of every one.