Protobuffers Are Wrong (2018) • Hacker News Board

Re: Protobuffers Are Wrong (2018)

#231

Post by [deleted] » Sat, Sep 06, 2025, 4:20 PM UTC

motorest wrote: post #217

lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

> Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. What I dislike the most about blog posts like this is that, although the blogger is very opinionated and critical of many things, the post dates back to 2018, protobuf is still dominant, and apparently during all these years the blogger failed to put together something that they felt was a bett…

[deleted]

Re: Protobuffers Are Wrong (2018)

#232

Post by bilkow » Sat, Sep 06, 2025, 4:40 PM UTC

sdenton4 wrote: post #224

rkagerer wrote: post #175

Earlier quoted context omitted.

More direct link to the juicy bit: https://github.com/stepchowfun/typical?tab=readme-ov-file#as... An asymmetric field in a struct is considered required for the writer, but optional for the reader.

That's a nice idea... But I believe the design direction of proto buffers was to make everything `optional`, because `required` tends to bite you later when you realize it should actually be optional.

My understanding is that asymmetric fields provide a migration path in case that happens, as stated in the docs:

> Unlike optional fields, an asymmetric field can safely be promoted to required and vice versa.

> [...]

> Suppose we now want to remove a required field. It may be unsafe to delete the field directly, since then clients might stop setting it before servers can handle its absence. But we can demote it to asymmetric, which forces servers to consider it optional and handle its potential absence, even though clients are still required to set it. Once that change has been rolled out (at least to servers), we can confidently delete the field (or demote it to optional), as the servers no longer rely on it.

Re: Protobuffers Are Wrong (2018)

#233

Post by b_e_n_t_o_n » Sat, Sep 06, 2025, 4:42 PM UTC

instig007 wrote: post #171

summerlight wrote: post #132

Earlier quoted context omitted.

If you see some statements like below on the serialization topic: > Make all fields in a message required. This makes messages product types. > One possible argument here is that protobuffers will hold onto any information present in a message that they don't understand. In principle this means that it's nondestructive to route a message through an intermediary that doesn't understand this version of its schema. Sure…

> If you see some statements like below on the serialization topic: > Make all fields in a message required. This makes messages product types. > Then it is fair to raise eyebrows on the author's expertise. It's fair to raise eyebrows on your expertise, since required fields don't contribute to b/w incompatibility at all, as every real-world protocol has a mandatory required version number that's tied to a direct par…

Is this satire?

Re: Protobuffers Are Wrong (2018)

#234

Post by akazantsev » Sat, Sep 06, 2025, 4:42 PM UTC

bithive123 wrote: post #137

I don't know if the author is right or wrong; I've never dealt with protobufs professionally. But I recently implemented them for a hobby project and it was kind of a game-changer. At some stage with every ESP or Arduino project, I want to send and receive data, i.e. telemetry and control messages. A lot of people use ad-hoc protocols or HTTP/JSON, but I decided to try the nanopb library. I ended up with a relatively…

You can also send JSON over UDP. Wiz smart bulbs do this for communication.

https://github.com/sbidy/pywizlight?tab=readme-ov-file#examp...

Re: Protobuffers Are Wrong (2018)

#235

Post by b_e_n_t_o_n » Sat, Sep 06, 2025, 4:46 PM UTC

btilly wrote: post #114

ndr wrote: post #3

Not even before the first line ends you get "They’re clearly written by amateurs". This is a rage bait, not worth the read.

The reasons for that line get at a fundamental tension. As David Wheeler famously said, "All problems in computer science can be solved by another level of indirection, except for the problem of too many indirections." Over time we accumulate cleverer and cleverer abstractions. And any abstraction that we've internalized, we stop seeing. It just becomes how we want to do things, and we have no sense of what cost we a…

I think you nailed it. So many complaints about Go for example basically come down to "it didn't let me create X abstraction" and that's basically the point.

Re: Protobuffers Are Wrong (2018)

#236

Post by b_e_n_t_o_n » Sat, Sep 06, 2025, 4:48 PM UTC

tptacek wrote: post #219

instig007 wrote: post #178

Earlier quoted context omitted.

Yeah, let's pretend that type algebra doesn't exist, and even if it does exist then it's not useful and definitely isn't practical in data protocols. Let's believe that the authors of protobuf considered everything, and since they aren't amateurs (by the virtue of having worked on protobuf at Google, presumably), every elaborated opinion that draws them as amateurs at applying type algebra in data protocol designs is…

They're not amateurs by virtue of being some of the most senior engineers ever to work at Google. You don't get to play the "ad hominem" card while calling them names. This whole thread is embarrassing.

[flagged]

Re: Protobuffers Are Wrong (2018)

#237

Post by orochimaaru » Sat, Sep 06, 2025, 4:56 PM UTC

lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

Protobufs aren’t new. They’re really just rpc over https. I’ve used dce-rpc in 1997 which had IDL. I believe CORBA used IDL as well although I personally did not use it. There have been other attempts like ejb, etc. which are pretty much the same paradigm.

The biggest plus with protobuf is the social/financial side and not the technology side. It’s open source and free from proprietary hacks like previous solutions.

Apart from that, distributed systems of which rpc is a sub topic are hard in general. So the expectation would be that it sucks.

Re: Protobuffers Are Wrong (2018)

#238

Post by tptacek » Sat, Sep 06, 2025, 4:59 PM UTC

instig007 wrote: post #208

xyzzyz wrote: post #176

Earlier quoted context omitted.

What you describe using many completely unnecessary mathematical terms is not only not found in “every real-world protocol”, but in fact is something virtually absent from overwhelming majority of actually used protocols, with a notable exception of the kind of protocol that gets a four digit numbered RFC document that describes it. Believe it or not, but in the software industry, nobody is defining a new “version nu…

> What you describe using many completely unnecessary mathematical terms Unnecessary for you, surely. > Believe it or not, but in the software industry, nobody is defining a new “version number” with “strictly defined algebra” when they want to add a new field to an communication protocol between two internal backend services. Name a protocol that doesn't have a respective version number, or without the defined algeb…

Instead of just handwaving about whether it's necessary or not, why not point to any protocol that relies on that attribute, and we can then evaluate how important that protocol is?

Re: Protobuffers Are Wrong (2018)

#239

Post by thinkharderdev » Sat, Sep 06, 2025, 5:04 PM UTC

iamdelirium wrote: post #142

thinkharderdev wrote: post #82

Earlier quoted context omitted.

If you need to deserialize an old version then it's not a problem. The unknown field is just ignored during deserialization. The problem is adding a required field since some clients might be sending the old value during the rollout. But in some situations you can be pretty confident that a field will be required always. And if you turn out to be wrong then it's not a huge deal. You add the new field as optional firs…

1. Then it's not really required if it can be ignored. 2. This is the problem, software (and protos) can live for a long time). They might be used by other clients elsewhere that you don't control. What you thought might not required 10 years down the line is not anymore. What you "think" is not a huge deal then becomes a huge deal and can cause downtime. 3. You're mixing business logic and over the wire field requir…

> Then it's not really required if it can be ignored.

It can be required in v2 but not in v1 which was my point. If the client is running v2 while the server is still on v1 temporarily, then there is no problem. The server just ignores the new field until it is upgraded.

> This is the problem, software (and protos) can live for a long time). They might be used by other clients elsewhere that you don't control. What you thought might not required 10 years down the line is not anymore. What you "think" is not a huge deal then becomes a huge deal and can cause downtime.

Part of this is just that trying to create a format that is suitable both as an rpc wire serialization format and ALSO a format suitable for long term storage leads to something that is not great for either use case. But even taking that into account, RDBMS have been dealing with this problem for decades and every RDBMS lets you define fields as non-nullable.

> If a message is required for an interface to function, you should be checking it anyway and returning the correct error. How is that change with proto supporting require?

That's my point, you have to do that check in code which clutters the implementation with validation noise. That and you often can't use the wire message in your internal domain model since you now have to do that defensive null-check everywhere the object is used.

Aside from that, protocol buffers are an interface definition language so should be able to encode some of the validation logic at least (make invalid states unrepresentable and all that). If you are just looking at the proto IDL you have no way of knowing whether a field is really required or not because there is no way to specify that.

Re: Protobuffers Are Wrong (2018)

#240

Post by thinkharderdev » Sat, Sep 06, 2025, 5:11 PM UTC

cherrycherry98 wrote: post #143

thinkharderdev wrote: post #48

Earlier quoted context omitted.

Yeah, as soon as you have a moderately complex type the generated code is basically useless. Honestly, ~80% of my gripes about protocol buffers could be alleviated by just allowing me to mark a message field as required.

Proto2 let you do this and the "required" keyword was removed because of the problems it introduces when evolving the schema in a system with many users that you don't necessarily control. Let's say you want to add a new required field, if your system receives messages from clients some clients may be sending you old data without the field and now the parse step fails because it detects a missing field. If you ever w…

> Philosophically, checking that a field is required or not is data validation and doesn't have anything to do with serialization

But protocol buffers is not just a serialization format it is an interface definition language. And not being able to communicate that a field is required or not is very limiting. Sometimes things are required to process a message. If you need to add a new field but be able to process older versions of the message where the field wasn't required (or didn't exist) then you can just add it as optional.

I understand that in some situations you have very hard compatibility requirements and it makes sense to make everything optional and deal with it in application code, but adding a required attribute to fields doesn't stop you from doing that. You can still just make everything optional. You can even add a CI lint that prevents people from merging code with required fields. But making required fields illegal at the interface definition level just strikes me as killing a fly with a bazooka.