Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

261–270 of 321 posts

Re: Protobuffers Are Wrong (2018)

#261

Post by sevensor »

vouwfietsman wrote: post #255
sevensor wrote: post #252

Earlier quoted context omitted.

I think I see what you’re getting at? My mental model is client and server, but you’re implying a more complex topology where no one service is uniquely a server or a client. You’d like to insert a new version at an arbitrary position in the graph without worrying about dependencies or the operational complexity of doing a phased deployment. The result is that you try to maintain a principled, constructive ambiguity…

Yes thats a big part, but even bigger is just the alignment of teams. Imagine team A building feature XYZ Team B is building TUV one of those features in each team deals with messages, the others are unrelated. At some point in time, both teams have to deploy. If you have to sync them up just to get the protocol to work, thats an extra complexity in the already complex work of the teams. If you can ignore this, great…

Ah, I see. Team A would like to deploy a new version of a service. It used to accept messages with schema S, but the new version accepts only S’ and not S. So the only thing you can do is define S’ so that it is ambiguous with S. Team B uses Team A’s service but doesn’t want to have to coordinate deployments with Team A.

I think the key source of my confusion was Team A not being able to continue supporting schema S once the new version is released. That certainly makes the problem harder.

Re: Protobuffers Are Wrong (2018)

#262

Post by cryptonector »

I've written several screeds in the comments here on HN about protobufs being terrible over the past few years. Basically the creators of PB ignored ASN.1 and built a bad version of mid-1980s ASN.1 and DER.

Tag-length-value (TLV) encodings are just overly verbose for no good reason. They are _NOT_ "self-describing", and one does not need everything tagged to support extensibility. Even where one does need tags, tag assignments can be fully automatic and need not be exposed to the module designer. Anyone with a modicum of time spent researching how ASN.1 handles extensibility with non-TLV encoding rules knows these things. The entire arc of ASN.1's evolution over two plus decades was all about extensibility and non-TLV encoding rules!

And yes, ASN.1 started with the same premise as PB, but 40 years ago. Thus it's terribly egregious that PB's designers did not learn any lessons at all from ASN.1!

Near as I can tell PB's designers thought they knew about encodings, but didn't, and near as I can tell they refused to look at ASN.1 and such because of the lack of tooling for ASN.1, but of course there was even less tooling for PB since it hadn't existed.

It's all exasperating.

Re: Protobuffers Are Wrong (2018)

#263

Post by cryptonector »

jnwatson wrote: post #32
lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

ASN.1 implements message versioning in an extremely precise way. Implementing a linter would be trivial.

This. Plus ASN.1 is pluggable as to encoding rules and has a large family of them:

  - BER/DER/CER (TLV)
  - OER and PER ("packed" -- no tags and
                 no lengths wherever
                 possible)
  - XER (XML!)
  - JER (JSON!)
  - GSER (textual representation)
  - you can add your own!
    (One could add one based on XDR,
     which would look a lot like OER/PER
     in a way.)
ASN.1 also gives you a way to do things like formalize typed holes.

Not looking at ASN.1, not even its history and evolution, when creating PB was a crime.

Re: Protobuffers Are Wrong (2018)

#264

Post by cryptonector »

tliltocatl wrote: post #181
bithive123 wrote: post #137

I don't know if the author is right or wrong; I've never dealt with protobufs professionally. But I recently implemented them for a hobby project and it was kind of a game-changer. At some stage with every ESP or Arduino project, I want to send and receive data, i.e. telemetry and control messages. A lot of people use ad-hoc protocols or HTTP/JSON, but I decided to try the nanopb library. I ended up with a relatively…

Embedded/constrained UDP is where protobuf wire format (but not google's libraries) rocks: IoT over cellular and such, where you need to fit everything into a single datagram (number of roundtrips is what determines power consumption). As to those who say "UDP is unreliable" - what you do is you implement ARQ on the application level. Just like TCP does it, except you don't have to waste roundtrips on SYN-SYN-ACK han…

> Yes, ASN is supposed to be the right solution - but there is no full-featured implementation that doesn't cost $$$$ and the whole thing is just too damn bloated.

Oh for crying out loud! PB had ZERO tooling available when it was created! It would have been much easier to create ASN.1 tooling w/ OER/PER and for some suitable subset of ASN.1 in 2001 that it was to a) create an IDL, b) create an encoding, and c) write tooling for N programming languages.

In fact, one thing one could have done is write a transpiler from the IDL to an AST that does all linting, analysis, and linking, and which one can then use to drive codegen for N languages. Or even better: have the transpiler produce a byte-coded representation of the modules and then for each programming language you only need to codegen the types but not the codecs -- instead for each language you need only write the interpreter for the byte-coded modules. I know because I've extended and maintained an [open source] ASN.1 compiler that fucking does [some of] these things.

Stop spreading this idea that ASN.1 is bloated. It's not. You can cut it down for your purposes. There's only 4 specifications for the language itself, of which the base one (x.680) is enough for almost everything (the others, X.681, X.682, and X.683, are mainly for parameterized types and formal typed hole specifications [the ASN.1 "information object system], which are awesome but you can live without). And these are some of the best-written and most-readable specifications ever written by any standards development organization -- they are a great gift from a few to all of mankind.

Re: Protobuffers Are Wrong (2018)

#265

Post by cryptonector »

nicce wrote: post #60
rapsey wrote: post #34

Earlier quoted context omitted.

There are none, protobufs are great.

Depends. ASN.1 is a beast and another industry standard, but unfortunately the best tooling is closed source.

There was ZERO PB tooling in 2000. Just write it for ASN.1 instead.

Re: Protobuffers Are Wrong (2018)

#266

Post by cryptonector »

theamk wrote: post #126
jandrese wrote: post #117

Earlier quoted context omitted.

"Those who cannot remember the past are condemned to repeat it" -- George Santayana

Oh, I remember ASN.1 very well, and I would not want to repeat it again. Protobufs have lots of problems, but at least they are better than ASN.1!

Details please.

Things people say who know very little about ASN.1:

- it's bloated! (it's not)

- it's had lots of vulnerabilities! (mainly in hand-coded codecs)

- it's expensive (it's not -- it's free and has been for two decades)

- it's ugly (well, sure, but so is PB's IDL)

- the language is context-dependent, making it harder to write a parser for (this is quite true, but so what, it's not that big a deal)

The vulnerabilities were only ever in implementations, and almost entirely in cases of hand-coded codecs, and the thing that made many of these vulnerabilities possible was the use of tag-length-value encoding rules (BER/DER/CER) which, ironically, Protocol Buffers bloody is too.

If you have a different objections to ASN.1, please list them.

Re: Protobuffers Are Wrong (2018)

#267

Post by cryptonector »

nicce wrote: post #55
mountainriver wrote: post #11

> Protobuffers correspond to the data you want to send over the wire, which is often related but not identical to the actual data the application would like to work with This sums up a lot of the issues I’ve seen with protobuf as well. It’s not an expressive enough language to be the core data model, yet people use it that way. In general, if you don’t have extreme network needs, then protobuf seems to cause more har…

On the other hand, ASN.1 is very expressive and can cover pretty much anything, but Protobuff was created because people thought ASN.1 is too complex. I guess we can't have both.

If people though ASN.1 was too big all they had to do was create a small profile of it large enough for the task at hand.

X.680 is fairly small. Require AUTOMATIC TAGs, remove manual tagging, remove REAL and EMBEDDED PDV and such things, and what you're left with is pretty small.

Re: Protobuffers Are Wrong (2018)

#268

Post by tliltocatl »

cryptonector wrote: post #264
tliltocatl wrote: post #181

Earlier quoted context omitted.

Embedded/constrained UDP is where protobuf wire format (but not google's libraries) rocks: IoT over cellular and such, where you need to fit everything into a single datagram (number of roundtrips is what determines power consumption). As to those who say "UDP is unreliable" - what you do is you implement ARQ on the application level. Just like TCP does it, except you don't have to waste roundtrips on SYN-SYN-ACK han…

> Yes, ASN is supposed to be the right solution - but there is no full-featured implementation that doesn't cost $$$$ and the whole thing is just too damn bloated. Oh for crying out loud! PB had ZERO tooling available when it was created! It would have been much easier to create ASN.1 tooling w/ OER/PER and for some suitable subset of ASN.1 in 2001 that it was to a) create an IDL, b) create an encoding, and c) write…

> It would have been much easier to create ASN.1 tooling w/ OER/PER and for some suitable subset of ASN.1 in 2001

Just by looking at your past comments - I agree that if google reused ASN.1, we would have lived in a better world. But the sad reality now is that PB gots tons of FOSS tooling and ASN.1 barely any (is there any free embedded-grade implementation other than asn1cc?) and figuring out what features you can use without having to pledge your kidney and soul to Nokalva is a bit hard.

I tried playing with ASN.1 before settling on protobuf. Don't recall which compiler I used, but immediately figured out that apparently datetime datatype is not supported, and the generated C code was bloated mess (so is google's protobuf - but not nanopb). Protobuf, on the other hand, was quite straightforward on what is and is not supported. So us mortals who aren't google and have a hard time justifying writing serdes from scratch gotta use what's available.

> Stop spreading this idea that ASN.1 is bloated

"Bloated" might be the wrong word - but it is large and it's damn hard for someone designing a new application to figure out which part is safe to use, because most sources focus on using it for decoding existing protocols.

Re: Protobuffers Are Wrong (2018)

#269

Post by grogers »

tliltocatl wrote: post #181
bithive123 wrote: post #137

I don't know if the author is right or wrong; I've never dealt with protobufs professionally. But I recently implemented them for a hobby project and it was kind of a game-changer. At some stage with every ESP or Arduino project, I want to send and receive data, i.e. telemetry and control messages. A lot of people use ad-hoc protocols or HTTP/JSON, but I decided to try the nanopb library. I ended up with a relatively…

Embedded/constrained UDP is where protobuf wire format (but not google's libraries) rocks: IoT over cellular and such, where you need to fit everything into a single datagram (number of roundtrips is what determines power consumption). As to those who say "UDP is unreliable" - what you do is you implement ARQ on the application level. Just like TCP does it, except you don't have to waste roundtrips on SYN-SYN-ACK han…

Other than ASN.1 PER, is there any other widely used encoding format that isn't self-describing? Using TLV certainly adds flexibility around schema evolution, but I feel like collectively we are wasting a fair amount of bytes because of it...

Re: Protobuffers Are Wrong (2018)

#270

Post by BobbyTables2 »

Even the low level implementation of protobuffers is pretty uninspiring.

Adds a lot of space overhead, specially for structs only used one yet not self descriptive either.

Doesn’t solve a lot of problems related to changes either.

Quite frankly, too many are using up in it because it came from Google and is supposed to be some sort of divinely inspired thing.

JSON, ASN.1, and even rigid C structs start to look a lot better.

Post reply on HN