Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

311–320 of 321 posts

Re: Protobuffers Are Wrong (2018)

#311
post #296

Earlier quoted context omitted.

The people who wrote PB clearly knew ASN.1. It was the most famous IDL at the time. Do you assume they just came one morning and decided to write PB without taking a look at what existed? Anyway, as stated PB does more than ASN.1. It specifies both the description format and the encoding. PB is ready to be used out of the box. You have a compact IDL and a performant encoding format without having to think about anyth…

> The people who wrote PB clearly knew ASN.1. And your assumption is based on what exactly? > It was the most famous IDL at the time. Strange that at the same time (2001) people were busy implementing everyting in Java and XML, not ASN.1 > Do you assume they just came one morning and decided to write PB without taking a look at what existed? Yes, that is a great assumption. Looking at what most companies do, this is…

> Strange that at the same time (2001) people were busy implementing everyting in Java and XML, not ASN.1

Yes. Meanwhile Google was designing an IDL with a default binary serialisation format. And this is not 2025 typical big corp, over staffed, fake HR levels heavy Google we are talking about. That’s Google in its heyday. I think you have answered your own comment.

Re: Protobuffers Are Wrong (2018)

#312
post #232

Earlier quoted context omitted.

That's a nice idea... But I believe the design direction of proto buffers was to make everything `optional`, because `required` tends to bite you later when you realize it should actually be optional.

My understanding is that asymmetric fields provide a migration path in case that happens, as stated in the docs: > Unlike optional fields, an asymmetric field can safely be promoted to required and vice versa. > [...] > Suppose we now want to remove a required field. It may be unsafe to delete the field directly, since then clients might stop setting it before servers can handle its absence. But we can demote it to a…

....or we can just say that everything is optional always, and leave it to the servers instead of the protocol to handle irregularities.

Re: Protobuffers Are Wrong (2018)

#313
post #126

Earlier quoted context omitted.

Oh, I remember ASN.1 very well, and I would not want to repeat it again. Protobufs have lots of problems, but at least they are better than ASN.1!

Details please. Things people say who know very little about ASN.1: - it's bloated! (it's not) - it's had lots of vulnerabilities! (mainly in hand-coded codecs) - it's expensive (it's not -- it's free and has been for two decades) - it's ugly (well, sure, but so is PB's IDL) - the language is context-dependent, making it harder to write a parser for (this is quite true, but so what, it's not that big a deal) The vuln…

Neither of those, the main problems are:

- There is no backward or forward compatibility by default.

(Sure, you can have every SEQUENCE have all fields OPTIONAL and ... at the end, but how many real-life schemas like that you have seen? Almost every ASN.1 you can find on the internet is static SEQUENCE, with no extensibility whatsoever)

- Tools are bad.

Yes, protoc can be a PITA to integrate into build system, but at least it (1) exists, (2) well-tested (3) supports many languages. Compared to ASN.1 where the good tooling is so rare, people routinely manually parse/generate the files!

- Honorable mention: using "tag" in TLV to describe only the type and not field name - that SEQUENCE(30) tag will be all over the place, and the contents will be wildly different. Compare to protobuf, where the "tag" is field index, and that's exactly what allows such a great forward/backward compatibility.

(Could ASN.1 fix those problems? Not sure. Yes, maybe one could write better tooling, but all the existing users know that extensibility is for the weak, and non-optional SEQUENCEs are the way to go. It is easier to write all-new format than try to change existing conventions.)

Re: Protobuffers Are Wrong (2018)

#314

Earlier quoted context omitted.

Protobufs are better but not best. Still, by far, the easiest thing to use and the safest is actual APIs. Like, in your application. Interfaces and stuff. Obviously if your thing HAS to communicate over the network that's one thing, but a lot of applications don't. The distributed system micro service stuff is a choice. Guys, distributed systems are hard. The extremely low API visibility combined with fragile network…

> Protobufs are better but not best. This sort of comments doesn't add anything to the discussion unless you are able to point out what you believe to be the best. It reads as an unnecessary and unsubstantiated put-down.

I... did.

Re: Protobuffers Are Wrong (2018)

#315
post #313

Earlier quoted context omitted.

Details please. Things people say who know very little about ASN.1: - it's bloated! (it's not) - it's had lots of vulnerabilities! (mainly in hand-coded codecs) - it's expensive (it's not -- it's free and has been for two decades) - it's ugly (well, sure, but so is PB's IDL) - the language is context-dependent, making it harder to write a parser for (this is quite true, but so what, it's not that big a deal) The vuln…

Neither of those, the main problems are: - There is no backward or forward compatibility by default. (Sure, you can have every SEQUENCE have all fields OPTIONAL and ... at the end, but how many real-life schemas like that you have seen? Almost every ASN.1 you can find on the internet is static SEQUENCE, with no extensibility whatsoever) - Tools are bad. Yes, protoc can be a PITA to integrate into build system, but at…

> - There is no backward or forward compatibility by default.

ASN.1 in 1984 had it. Later ASN.1 evolved to have a) explicit extensibility markers, and b) the `EXTENSIBILITY IMPLIED` module option that implies every SEQUENCE, SET, ENUM, and other things are extensible by default, as if they ended in `, ...`.

There are good reasons for this change:

- not all implementors had understood the intent, so not all had implemented "ignore unexpected new fields"

- sometimes you want non-extensible things

- you may actually want to record in the syntax all the sets of extensions

> - Tools are bad.

But there were zero -ZERO!- tools for PB when Google created PB. Don't you see that "the tools that existed were shit" is not a good argument for creating tools for a completely new thing instead?

> - Honorable mention: using "tag" in TLV to describe only the type and not field name - that SEQUENCE(30) tag will be all over the place, and the contents will be wildly different. Compare to protobuf, where the "tag" is field index, and that's exactly what allows such a great forward/backward compatibility.

In a TLV encoding you can very much use the "type" as the tag for every field sometimes, namely when there would be no ambiguity due to OPTIONAL fields being present or absent, and when you do have such ambiguities you can resort to manual tagging with field numbers or whatever you want. For example:

  Thing ::= SEQUENCE {
    a UTF8String,
    b UTF8String
  }
works even though both fields get the same tag (when using a TLV encoding) because both fields are required, while this is broken:

  Broken ::= SEQUENCE {
    a UTF8String OPTIONAL,
    b UTF8String
  }
and you would have to fix it with something like:

  Fixed ::= SEQUENCE {
    a [0] UTF8String OPTIONAL,
    b UTF8String
  }
What PB does is require the equivalent of manually applying what ASN.1 calls IMPLICIT tags to every field, which is silly and makes it harder to decode data w/o reference to the module that defines its schema (this last is sketchy anyways, and I don't think it is a huge advantage for the ASN.1 BER/DER way of doing things, though others will disagree).

> (Could ASN.1 fix those problems? Not sure. Yes, maybe one could write better tooling, but all the existing users know that extensibility is for the weak, and non-optional SEQUENCEs are the way to go. It is easier to write all-new format than try to change existing conventions.)

ASN.1 does not have these problems.

Better tooling does exist and can exist -- it's no different than writing PB tooling, at least for a subset of ASN.1, because ASN.1 does have many advanced features that PB lacks, and obviously implementing all of ASN.1 is more work than implementing all of PB.

> It is easier to write all-new format than try to change existing conventions.

Maybe, but only if you have a good handle on what came before.

I strongly recommend that you actually read x.680.

Re: Protobuffers Are Wrong (2018)

#316
post #256
post #133

Earlier quoted context omitted.

Yeah, I’d probably say something more like, “we leverage protobuf built ins to make a slightly more advanced back compat system” We do rename deprecated fields and often give new fields their names. We rely on the field number to make that work.

> We do rename deprecated fields and often give new fields their names. We rely on the field number to make that work. Why share names? Wouldn't it be safer to, well, not?

The code becomes hard to read. You might need to change int health to float health. In that case “health” properly describes the idea. We’d change this to int DEPRECATED_health and float health.

Folks can argue that’s ugly but I’ve not seen one instance of someone confused.

Re: Protobuffers Are Wrong (2018)

#317

Earlier quoted context omitted.

Seems like a lot of effort to avoid adding a message version field. I’m not a web guy, so maybe I’m missing the point here, but I always embed a schema version field in my data.

I get that. The point is that its hard to prevent asymmetry in message versions if you are working with many communicating systems. Lets say four services inter-communicate with some protocol, it is extremely annoying to impose a deployment order where the producer of a message type is the last to upgrade the message schema, as this causes unnecessary dependencies between the release trains of these services. At the…

> one cannot simply say: "I don't know this message version, I will disregard it" because in live systems this will mean the systems go out of sync, data is lost, stuff breaks, etc.

You already need to deal with lost messages, rejected messages, so just treat this case the same. If you have versions surely you have code to deal with mismatches and e.g. fail back to the older version.

Re: Protobuffers Are Wrong (2018)

#318

Earlier quoted context omitted.

Seems like a lot of effort to avoid adding a message version field. I’m not a web guy, so maybe I’m missing the point here, but I always embed a schema version field in my data.

I get that. The point is that its hard to prevent asymmetry in message versions if you are working with many communicating systems. Lets say four services inter-communicate with some protocol, it is extremely annoying to impose a deployment order where the producer of a message type is the last to upgrade the message schema, as this causes unnecessary dependencies between the release trains of these services. At the…

> Lets say four services inter-communicate with some protocol, it is extremely annoying to impose a deployment order where the producer of a message type is the last to upgrade the message schema

i don't know how you arrived at this conclusion

the protocol is the unifying substrate, it is the source of truth, the services are subservient to the protocol, it's not the other way around

also it's not just like each service has a single version, each instance of each service can have separate versions as well!

what you're describing as "annoying" is really just "reality", you can't hand-wave away the problems that reality presents

Re: Protobuffers Are Wrong (2018)

#319

Earlier quoted context omitted.

If there are errors in implementation of general constructs, they tend to be visible at their every use, and get rapidly fixed. Some general constructs are better than the others, because they have an algebraic theory behind them, and sometimes that theory was already researched for a few hundred years. For example, product/coproduct types mentioned in the article are quite close to addition and multiplication that w…

Hm, that's idealistic... I've certainly run into cases where small changes in general systems led to hard-to-detect bugs, which took a great deal of investigation to figure out. Not all failures are catastrophic. The technology is quite alive, which is why it hasn't been 'fixed' - changing the wheels on a moving car, and all that. The actual disappointment is that a better alternative hasn't taken off in the six year…

That's not idealistic, that's how arithmetics work. If you use the same generic thing more times, you have the higher chance of discovering it broken. The fact that you've run into cases means that chance is never zero, and is irrelevant to the discussion.

As was already mentioned in the article, PB solve a problem that likely only Google has, even if that. State of the art nowadays is JSON/JSONL. If it grows too large, gzip it.

When someone is using third-party closed proprietary technologies to be "not like the rest", it usually doesn't work that well for their business.

The technology is "alive" until it didn't follow the path of Closure, GWT, and the rest of "we use it on the most loaded page of the world" technology. PB will be on the same graveyard soon.

Re: Protobuffers Are Wrong (2018)

#320
I get the author's points, and they all are valid, but I don't understand why people would use generated code and its types through their entire project.

This is just asking for trouble when the API will inevitably break as all APIs will do eventually. In our projects I mandated and pushed really hard that we create intermediary data classes that correspond one to one to the protobufs (at first).

I got a lot of angry faces and reactions in PR due to the seemingly useless boiler plate code required but it saved our butts so many times when the API changed just before a release that it became the de facto standard.

Also, protobufs and GRPCs are a de facto standards. Are there better alternatives? Yes. Should you use those? Most likely not because the point of serialization frameworks is to be used by many people in various tech stacks.

Post reply on HN