Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

271–280 of 321 posts

Re: Protobuffers Are Wrong (2018)

#271

Post by wnoise »

ericpauley wrote: post #63

I lost the plot here when the author argued that repeated fields should be implemented as in the pure lambda calculus... Most of the other issues in the article can be solved be wrapping things in more messages. Not great, not terrible. As with the tightly-coupled issues with Go, I'll keep waiting for a better approach any decade now. In the meantime, both tools (for their glaring imperfections) work well enough, sol…

They didn't. Pure lambda calculus would have been "a function that when applied to a number encoded as a function, extracts that value".

They did it essentially as a linked list, C-strings, or UTF-8 characters: "current data, and is there more (next pointer, non-null byte, continuation bit set)?" They also noted that it could have this semantics without necessarily following this implementation encoding, though that seems like a dodge to me; length-prefixed array is a perfectly fine primitive to have, and shouldn't be inferred from something that can map to it.

Re: Protobuffers Are Wrong (2018)

#272

Post by dlahoda »

lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

https://github.com/dfinity/candid/blob/master/spec/Candid.md

Re: Protobuffers Are Wrong (2018)

#273

Post by lostdog »

kentonv wrote: post #253
lostdog wrote: post #230

Earlier quoted context omitted.

It gets super frustrating to have to empty/null check fields everywhere you use them, especially for fields that are effectively required for the message to make sense. A very common example I see is Vec3 (just x, y, z). In proto2 you should be checking for the presence of x,y,z every time you use them, and when you do that in math equations, the incessant existence checks completely obscure the math. Really, you wan…

> It gets super frustrating to have to empty/null check fields everywhere you use them, especially for fields that are effectively required for the message to make sense. That's why Protobuf and Cap'n Proto have default values. You should not bother checking for presence of fields that are always supposed to be there. If the sender forgot to set a field, then they get the default value. That's their problem. > just a…

Sadly, the default values are an even bigger source of bugs. We just caught another one at $work where a field was never being filled in, but the default values made it look fine. It caused hidden failures later on.

It's an incredibly frustrating "feature" to deal with, and causes lots of problems in proto3.

Re: Protobuffers Are Wrong (2018)

#274

Post by OrangeDelonge »

sethammons wrote: post #167
BugsJustFindMe wrote: post #14

I went into this article expecting to agree with part of it. I came away agreeing with all of it. And I want to point out that Go also shares some of these catastrophic data decisions (automatic struct zero values that silently do the wrong thing by default).

We got bit by a default value in a DMS task where the target column didn't exist so the data wasn't replicated and the default value was "this work needs to be done." This is not pb nor go. A sensible default of invalid state would have caught this. So would an error and crash. Either would have been better than corrupt data.

You mean aws dms insterted the string literal “this work needs to be done” into your db?

Re: Protobuffers Are Wrong (2018)

#275

Post by cryptonector »

tliltocatl wrote: post #268
cryptonector wrote: post #264

Earlier quoted context omitted.

> Yes, ASN is supposed to be the right solution - but there is no full-featured implementation that doesn't cost $$$$ and the whole thing is just too damn bloated. Oh for crying out loud! PB had ZERO tooling available when it was created! It would have been much easier to create ASN.1 tooling w/ OER/PER and for some suitable subset of ASN.1 in 2001 that it was to a) create an IDL, b) create an encoding, and c) write…

> It would have been much easier to create ASN.1 tooling w/ OER/PER and for some suitable subset of ASN.1 in 2001 Just by looking at your past comments - I agree that if google reused ASN.1, we would have lived in a better world. But the sad reality now is that PB gots tons of FOSS tooling and ASN.1 barely any (is there any free embedded-grade implementation other than asn1cc?) and figuring out what features you can…

For sure PB is a fact of life now. A regrettable fact of life, but perhaps a lesson (that few will heed).

Re: Protobuffers Are Wrong (2018)

#276

Post by naikrovek »

lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

TLV style binary formats are all you need. The “Type” in that acronym is a 32-bit number which you can use to version all of your stuff so that files are backwards compatible. Software that reads these should read all versions of a particular type and write only the latest version.

Code for TLV is easy to write and to read, which makes viewing programs easy. TLV data is fast for computers to write and to read.

Protobuf is overused because people are fucking scared to death to write binary data. They don’t trust themselves to do it, which is just nonsense to me. It’s easy. It’s reliable. It’s fast.

Re: Protobuffers Are Wrong (2018)

#277

Post by blarus78 »

ziml77 wrote: post #153
klodolph wrote: post #96

Protobuffers suck as a core data model. My take? Use them as a serialization and interchange format, nothing more. > This puts us in the uncomfortable position of needing to choose between one of three bad alternatives: I don’t think there is a good system out there that works for both serialization and data models. I’d say it’s a mostly unsolved problem. I think I am happy with protobufs. I know that I have to fight…

> My take? Use them as a serialization and interchange format, nothing more. Isn't that exactly what they're intended for? I'm confused how anyone would even think to use them any other way.

Agreed, it's interesting to see so many people complaining when they are just misunderstanding / misusing protobufs entirely. Sure the implementation could be better but it's not a huge problem.

Re: Protobuffers Are Wrong (2018)

#278

Post by vouwfietsman »

sevensor wrote: post #261
vouwfietsman wrote: post #255

Earlier quoted context omitted.

Yes thats a big part, but even bigger is just the alignment of teams. Imagine team A building feature XYZ Team B is building TUV one of those features in each team deals with messages, the others are unrelated. At some point in time, both teams have to deploy. If you have to sync them up just to get the protocol to work, thats an extra complexity in the already complex work of the teams. If you can ignore this, great…

Ah, I see. Team A would like to deploy a new version of a service. It used to accept messages with schema S, but the new version accepts only S’ and not S. So the only thing you can do is define S’ so that it is ambiguous with S. Team B uses Team A’s service but doesn’t want to have to coordinate deployments with Team A. I think the key source of my confusion was Team A not being able to continue supporting schema S…

Exactly!

Re: Protobuffers Are Wrong (2018)

#279

Post by ardit33 »

motorest wrote: post #257
ardit33 wrote: post #251

Earlier quoted context omitted.

JSON exists, and when compressed it is pretty efficient. (not as efficient as protobuff though). For client facing protocol Protobufs is a nightmare to use. For Machine to Machine services, it is ok-ish, yet personally I still don't like it. When I was at Spotify we ditched it for client side apis (server to mobile/web), and never looked back. No one liked working with it.

> JSON exists (...) The blog post leads with the personal assertion that "ad-hoc and built by amateurs". Therefore I doubt that JSON, a data serialization language designed by trimming most of JavaScript out and to be parses with eval(), would meet the opinionated high bar. Also, JSON is a data interchange language, and has no support for types beyond the notoriously ill-defined primitives. In contrast, protobuf is a…

You must be young. XML and XML Schemas existed before JSON or Protobuf, and people ditched them for a good reason and JSON took over.

Protobuf is just another version of the old RPC/Java Beans, etc... of a binary format. Yes, it is more efficient data wise than JSON, but it is a PITA to work on and debug with.

Re: Protobuffers Are Wrong (2018)

#280

Post by tiew9Vii »

ziml77 wrote: post #153
klodolph wrote: post #96

Protobuffers suck as a core data model. My take? Use them as a serialization and interchange format, nothing more. > This puts us in the uncomfortable position of needing to choose between one of three bad alternatives: I don’t think there is a good system out there that works for both serialization and data models. I’d say it’s a mostly unsolved problem. I think I am happy with protobufs. I know that I have to fight…

> My take? Use them as a serialization and interchange format, nothing more. Isn't that exactly what they're intended for? I'm confused how anyone would even think to use them any other way.

Not specific to protobufs but a lot of people/projects especially if doing MVC, push the models in the API layer all the way down the stack and they become the domain, instead of having a loose coupling between the domain and serialization format. In the old days we used to have DTO's for separation but they went out of fashion.
Post reply on HN