Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

171–180 of 321 posts

Re: Protobuffers Are Wrong (2018)

#171

Post by instig007 »

summerlight wrote: post #132
missinglugnut wrote: post #128

Earlier quoted context omitted.

>Most of the complexity of serialization comes from implementation compatibility between different timepoints. The author talks about compatibility a fair bit, specifically the importance of distinguishing a field that wasn't set from one that was intentionally set to a default, and how protobuffs punted on this. What do you think they don't understand?

If you see some statements like below on the serialization topic: > Make all fields in a message required. This makes messages product types. > One possible argument here is that protobuffers will hold onto any information present in a message that they don't understand. In principle this means that it's nondestructive to route a message through an intermediary that doesn't understand this version of its schema. Sure…

> If you see some statements like below on the serialization topic:

> Make all fields in a message required. This makes messages product types.

> Then it is fair to raise eyebrows on the author's expertise.

It's fair to raise eyebrows on your expertise, since required fields don't contribute to b/w incompatibility at all, as every real-world protocol has a mandatory required version number that's tied to a direct parsing strategy with strictly defined algebra, both for shrinking (removing data fragments) and growing (introducing data fragments) payloads. Zero-values and optionality in protobuf is one version of that algebra, it's the most inferior one, subject to lossy protocol upgrades, and is the easiest one for amateurs to design. Then, there's next lavel when the protocol upgrade is defined in terms of bijective functions and other elements of symmetric groups that can tell you whether the newly announced data change can be carried forward (new required field) or dropped (removed field) as long as both the sending and receiving ends are able to derive new compound structures from previously defined pervasive types (the things the protobuf says are oneoffs and messages, for example).

Re: Protobuffers Are Wrong (2018)

#172

Post by giveita »

nice_byte wrote: post #123

> Make all fields in a message required. funnily enough, this line alone reveals the author to be an amateur in the problem space they are writing so confidently about.

I sniffed this. I am not familiar with protobufs, but aware they are for efficiency on the wire. The fact he only really talks about type systems and not the before vs. after of the affect on the wire was disappointing but also made me suspect to if this was a good piece.

Re: Protobuffers Are Wrong (2018)

#173

Post by xyzzyz »

Granted, on paper it’s a cool feature. But I’ve never once seen an application that will actually preserve that property.

Chances are, the author literally used software that does it as he wrote these words. This feature is critical to how Chrome Sync works. You wouldn’t want to lose synced state if you use an older browser version on another device that doesn’t recognize the unknown fields and silently drops them. This is so important that at some point Chrome literally forked protobuf library so that unknown fields are preserved even if you are using protobuf lite mode.

Re: Protobuffers Are Wrong (2018)

#175

Post by rkagerer »

jitl wrote: post #19
lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

Not widely used but I like Typical's approach https://github.com/stepchowfun/typical > Typical offers a new solution ("asymmetric" fields) to the classic problem of how to safely add or remove fields in record types without breaking compatibility. The concept of asymmetric fields also solves the dual problem of how to preserve compatibility when adding or removing cases in sum types.

More direct link to the juicy bit: https://github.com/stepchowfun/typical?tab=readme-ov-file#as...

An asymmetric field in a struct is considered required for the writer, but optional for the reader.

Re: Protobuffers Are Wrong (2018)

#176

Post by xyzzyz »

instig007 wrote: post #171
summerlight wrote: post #132

Earlier quoted context omitted.

If you see some statements like below on the serialization topic: > Make all fields in a message required. This makes messages product types. > One possible argument here is that protobuffers will hold onto any information present in a message that they don't understand. In principle this means that it's nondestructive to route a message through an intermediary that doesn't understand this version of its schema. Sure…

> If you see some statements like below on the serialization topic: > Make all fields in a message required. This makes messages product types. > Then it is fair to raise eyebrows on the author's expertise. It's fair to raise eyebrows on your expertise, since required fields don't contribute to b/w incompatibility at all, as every real-world protocol has a mandatory required version number that's tied to a direct par…

What you describe using many completely unnecessary mathematical terms is not only not found in “every real-world protocol”, but in fact is something virtually absent from overwhelming majority of actually used protocols, with a notable exception of the kind of protocol that gets a four digit numbered RFC document that describes it. Believe it or not, but in the software industry, nobody is defining a new “version number” with “strictly defined algebra” when they want to add a new field to an communication protocol between two internal backend services.

Re: Protobuffers Are Wrong (2018)

#177

Post by porridgeraisin »

usrnm wrote: post #168
leoc wrote: post #124

Earlier quoted context omitted.

What about Cap’n Proto https://capnproto.org/ ? (Don't know much about these things myself, but it's a name that usually comes up in these discussions.)

Cap'n'proto is not very nice to work with in C++, and I'd discourage anyone from using it from other programming languages, the implementations are just not there yet. We use both cnp and protobufs at work, and I vastly prefer protobufs, even for C++. I only wish they stayed the hell away from abseil, though.

I always thought people had a positive view on abseil, never used it myself other than when tinkering on random projects. What's the main issue?

Re: Protobuffers Are Wrong (2018)

#178

Post by instig007 »

jilles wrote: post #8
ndr wrote: post #3

Not even before the first line ends you get "They’re clearly written by amateurs". This is a rage bait, not worth the read.

The best way to get your point across is by starting with ad-hominem attacks to assert your superior intelligence.

Yeah, let's pretend that type algebra doesn't exist, and even if it does exist then it's not useful and definitely isn't practical in data protocols. Let's believe that the authors of protobuf considered everything, and since they aren't amateurs (by the virtue of having worked on protobuf at Google, presumably), every elaborated opinion that draws them as amateurs at applying type algebra in data protocol designs is a personal ad-hominem attack.

Re: Protobuffers Are Wrong (2018)

#180

Post by porridgeraisin »

xyzzyz wrote: post #176
instig007 wrote: post #171

Earlier quoted context omitted.

> If you see some statements like below on the serialization topic: > Make all fields in a message required. This makes messages product types. > Then it is fair to raise eyebrows on the author's expertise. It's fair to raise eyebrows on your expertise, since required fields don't contribute to b/w incompatibility at all, as every real-world protocol has a mandatory required version number that's tied to a direct par…

What you describe using many completely unnecessary mathematical terms is not only not found in “every real-world protocol”, but in fact is something virtually absent from overwhelming majority of actually used protocols, with a notable exception of the kind of protocol that gets a four digit numbered RFC document that describes it. Believe it or not, but in the software industry, nobody is defining a new “version nu…

Yeah. And for anyone curious about the actual content hidden under the jargon-kludge-FP-nerd parent comment, here's my attempt at deciphering it.

They seem to be saying that you have to publish code that can change a type from schema A to schema B... And back, whenever you make a schema B. This is the "algebra". The "and back" part makes it bijective. You do this at the level of your core primitive types so that it's reused everywhere. This is what they meant by "pervasive" and it ties into the whole symmetric groups thing.

Finally, it seems like when you're making a lossy change, where a bijection isn't possible, they want you to make it incompatible. i.e, if you replaced address with city, then you cannot decode the message in code that expects address.

Post reply on HN