Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

131–140 of 321 posts

Re: Protobuffers Are Wrong (2018)

#131

Post by zigzag312 »

haberman wrote: post #99
zigzag312 wrote: post #79

Earlier quoted context omitted.

It looks similar to how vint64 lib encodes varints. Total length of varint can be determined via the first byte alone.

I advocated for PrefixVarint (which seems equivalent to vint64 ) for WebAssembly, but it was decided against, in favor of LEB128: https://github.com/WebAssembly/design/issues/601 The recent CREL format for ELF also uses the more established LEB128: https://news.ycombinator.com/item?id=41222021 At this point I don't feel like I have a clear opinion about whether PrefixVarint is worth it, compared with LEB128.

Just remember that XML was more established than JSON for a long time.

Re: Protobuffers Are Wrong (2018)

#132

Post by summerlight »

missinglugnut wrote: post #128
summerlight wrote: post #94

https://news.ycombinator.com/item?id=18190005 Just FYI: an obligatory comment from the protobuf v2 designer. Yeah, protobuf has lots of design mistakes but this article is written by someone who does not understand the problem space. Most of the complexity of serialization comes from implementation compatibility between different timepoints. This significantly limits design space.

>Most of the complexity of serialization comes from implementation compatibility between different timepoints. The author talks about compatibility a fair bit, specifically the importance of distinguishing a field that wasn't set from one that was intentionally set to a default, and how protobuffs punted on this. What do you think they don't understand?

If you see some statements like below on the serialization topic:

> Make all fields in a message required. This makes messages product types.

> One possible argument here is that protobuffers will hold onto any information present in a message that they don't understand. In principle this means that it's nondestructive to route a message through an intermediary that doesn't understand this version of its schema. Surely that's a win, isn't it?

> Granted, on paper it's a cool feature. But I've never once seen an application that will actually preserve that property.

Then it is fair to raise eyebrows on the author's expertise. And please don't ask if I'm attached to protobuf; I can roast the protocol buffer on its wrong designs for hours. It is just that the author makes series of wrong claims presumably due to their bias toward principled type systems and inexperience of working on large scale systems.

Re: Protobuffers Are Wrong (2018)

#133

Post by tyleo »

swiftcoder wrote: post #88
tyleo wrote: post #57

Earlier quoted context omitted.

We use protocol buffers on a game and we use the back compat stuff all the time. We include a version number with each release of the game. If we change a proto we add new fields and deprecate old ones and increment the version. We use the version number to run a series of steps on each proto to upgrade old fields to new ones.

> We use the version number to run a series of steps on each proto to upgrade old fields to new ones It sounds like you've built your own back-compat functionality on top of protobuf? The only functionality protobuf is giving you here is optional-by-default (and mandatory version numbers, but most wire formats require that)

Yeah, I’d probably say something more like, “we leverage protobuf built ins to make a slightly more advanced back compat system”

We do rename deprecated fields and often give new fields their names. We rely on the field number to make that work.

Re: Protobuffers Are Wrong (2018)

#134

Post by wvenable »

tgma wrote: post #108
xmddmx wrote: post #15

I share the author's sentiment. I hate these things. True story: trying to reverse engineer macOS Photos.app sqlite database format to extract human-readable location data from an image. I eventually figured it out, but it was: A base64 encoded Binary Plist format with one field containing a ProtoBuffer which contained another protobuffer which contained a unicode string which contained improperly encoded data (for e…

> This could have been a simple JSON string. There's nothing "simple" about parsing JSON as a serialization format.

Except that most often you can just look at it and figure it out.

Re: Protobuffers Are Wrong (2018)

#135

Post by taeric »

thinkharderdev wrote: post #93
taeric wrote: post #67

Earlier quoted context omitted.

To add to the sibling, I've seen this with Java enums a lot. People will add it so that the value is consumed using the enum as fast as they can. This works well as long as the value is not retrieved from data. As soon as you do that, you lose the ability to add to the possible values in a rolling release way. It can be very frustrating to know that we can't push a new producer of a value before we first change all c…

But this is something you should be able to handle on a case-by-case basis. If you have a type which is stored durably as protobuf then adding required fields is much harder. But if you are just dealing with transient rpc messages then it can be done relatively easily in a two step process. First you add the field as optional and then once all producers are upgraded (and setting the new field), make it required. It's…

Largely true. If you are at Google scale, odds are you have mixed fleets deployed. Such that it is a bit of involved process. But it is well defined and doable. I think a lot of us would rather not do a dance we don't have to do?

Re: Protobuffers Are Wrong (2018)

#137

Post by bithive123 »

I don't know if the author is right or wrong; I've never dealt with protobufs professionally. But I recently implemented them for a hobby project and it was kind of a game-changer.

At some stage with every ESP or Arduino project, I want to send and receive data, i.e. telemetry and control messages. A lot of people use ad-hoc protocols or HTTP/JSON, but I decided to try the nanopb library. I ended up with a relatively neat solution that just uses UDP packets. For my purposes a single packet has plenty of space, and I can easily extend this approach in the future. I know I'm not the first person to do this but I'll probably keep using protobufs until something better comes along, because the ecosystem exists and I can focus on the stuff I consider to be fun.

Re: Protobuffers Are Wrong (2018)

#138

Post by _zoltan_ »

bithive123 wrote: post #137

I don't know if the author is right or wrong; I've never dealt with protobufs professionally. But I recently implemented them for a hobby project and it was kind of a game-changer. At some stage with every ESP or Arduino project, I want to send and receive data, i.e. telemetry and control messages. A lot of people use ad-hoc protocols or HTTP/JSON, but I decided to try the nanopb library. I ended up with a relatively…

and since it's UDP, if it's lost it's lost. and since it's not standard http/JSON, nobody will have a clue in a year and can't decode it.

to learn and play with it it's fine, else why complicate life?

Re: Protobuffers Are Wrong (2018)

#139

Post by ryukoposting »

Protobuf's original sin was failing to distinguish zero/false from undefined/unset/nil. Confusion around the semantics of a zero value are the root of most proto-related bugs I've come across. At the same time, that very characteristic of protobuf makes its on-wire form really efficient in a lot of cases.

Nearly every other complaint is solved by wrapping things in messages (sorry, product types). Don't get the enum limitation on map keys, that complaint is fair.

Protobuf eliminates truckloads of stupid serialization/deserialization code that, in my embedded world, almost always has to be hand-written otherwise. If there was a tool that automatically spat out matching C, Kotlin, and Swift parsers from CDDL, I'd certainly give it a shot.

Re: Protobuffers Are Wrong (2018)

#140

Post by rednafi »

I like the problems that Protobuf solves, just not the way it solves them.

Protobuf as a language feels clunky. The “type before identifier” syntax looks ancient and Java-esque.

The tools are clunky too. protoc is full of gotchas, and for something as simple as validation, you need to add a zillion plugins and memorize their invocation flags.

From tooling to workflow to generated code, it’s full of Google-isms and can be awkward to use at times.

That said, the serialization format is solid, and the backward-compatibility paradigms are genuinely useful. Buf adds some niceties to the tooling and makes it more tolerable. There’s nothing else that solves all the problems Protobuf solves.

Post reply on HN