Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

151–160 of 321 posts

Re: Protobuffers Are Wrong (2018)

#151

Post by nomel »

dangets wrote: post #127
stickfigure wrote: post #115

Earlier quoted context omitted.

Backwards compatibility is just not an issue in self-describing structures like JSON, Java serialization, and (dating myself) Hessian. You can add fields and you can remove fields. That's enough to allow seamless migrations. It's only positional protocols that have this problem.

You can remove JSON fields at the cost of breaking your clients at runtime that expect those fields. Of course the same can happen with any deserialization libraries, but protobufs at least make it more explicit - and you may also be more easily able to track down consumers using older versions.

For the missing case, whenever I use json, I always start with a sane default struct, then overwrite those with the externally provided values. If a field is missing, it will be handled reasonably.

Re: Protobuffers Are Wrong (2018)

#152

Post by taeric »

rimunroe wrote: post #144
cherrycherry98 wrote: post #143

Earlier quoted context omitted.

Proto2 let you do this and the "required" keyword was removed because of the problems it introduces when evolving the schema in a system with many users that you don't necessarily control. Let's say you want to add a new required field, if your system receives messages from clients some clients may be sending you old data without the field and now the parse step fails because it detects a missing field. If you ever w…

> Philosophically, checking that a field is required or not is data validation and doesn't have anything to do with serialization. My issue is that people seem to like to use protobuf to describe the shape of APIs rather than just something to handle serialization. I think it's very bad at the describing API shapes.

I think it is somewhat of a natural failure of DRY taken to the extreme? People seem to want to get it so that they describe the API in a way that is then generated for clients and implementations.

It is amusing, in many ways. This is specifically part of what WSDL aspired to, but people were betrayed by the big companies not having a common ground for what shapes they would support in a description.

Re: Protobuffers Are Wrong (2018)

#153

Post by ziml77 »

klodolph wrote: post #96

Protobuffers suck as a core data model. My take? Use them as a serialization and interchange format, nothing more. > This puts us in the uncomfortable position of needing to choose between one of three bad alternatives: I don’t think there is a good system out there that works for both serialization and data models. I’d say it’s a mostly unsolved problem. I think I am happy with protobufs. I know that I have to fight…

> My take? Use them as a serialization and interchange format, nothing more.

Isn't that exactly what they're intended for? I'm confused how anyone would even think to use them any other way.

Re: Protobuffers Are Wrong (2018)

#154

Post by anonymousiam »

mattnewton wrote: post #36
lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

Exactly, I think of protobuffers like I think of Java or Go - at least they weren’t writing it in C++. Dragging your org away from using poorly specified json is often worth these papercuts IMO.

The original RPC code, from which Google derived their protobuf stuff was written in (pre-ANSI) C at Sun Microsystems.

Re: Protobuffers Are Wrong (2018)

#155

Post by tgma »

wvenable wrote: post #134
tgma wrote: post #108

Earlier quoted context omitted.

> This could have been a simple JSON string. There's nothing "simple" about parsing JSON as a serialization format.

Except that most often you can just look at it and figure it out.

Sure you can look at it[1], but you're not expected to look at Apple Photos database. The computer is.

Write a correct JSON parser, compare with protobuf on various metrics, and then we can talk.

[1]: although to be fair, I am older than kids whose first programming language was JavaScript, so I do not think of JSON object format with property names in quotes and integers that need to be wrapped as strings to be safe, etc., lack of comma after the last entry--to be fair this last one is a problem in writing, not reading JSON--as the most natural thing

Re: Protobuffers Are Wrong (2018)

#156

Post by tgma »

xmddmx wrote: post #91
fluoridation wrote: post #64

Earlier quoted context omitted.

I mean... you can nest-encode stuff in any serial format. You're not describing a problem either intrinsic or unique to Protobuf, you're just seeing the development org chart manifested into a data structure.

Good points this wasn't entirely a protobuf-specific issue, so much as it was a (likely hierarchical and historical set of) bad decisions to use it at all. Using Protobuffers for a few KB of metadata, when the photo library otherwise is taking multiple GB of data, is just pennywise pound foolish. Of course, even my preference for a simple JSON string would be problematic: data in a database really should be stored pr…

You have to take into account the fact that iPhoto app has had many iterations. The binary plist stuff is very likely the native NSArchive "object archiving (serialization)" that is done by Obj-C libraries. They probably started using protobuf at some point later after iCloud. I suspect the unicode crap you are facing may even predate Cocoaization of the app (they probably used Carbon API).

So it would make it a set of historical decisions, but I am not convinced they are necessarily bad decisions given the constraints. Each layer is likely responsible for handing edge cases in the application that you and I are not privy to.

Re: Protobuffers Are Wrong (2018)

#157

Post by vl »

rimunroe wrote: post #23
taeric wrote: post #2

I'm more than a little curious what event caused such a strong objection to protobuffers. :D I do tend to agree that they are bad. I also agree that people put a little too much credence in "came from Google." I can't bring myself to have this much anger towards it. Had to have been something that sparked this.

I'm just a frontend developer so most of my exposure is just as an API consumer and not someone working on the service side of things. That said: A few years ago I moved to a large company where protobufs were the standard way APIs were defined. When I first started working with the generated TypeScript code, I was confused as to why almost all fields on generated object types were marked as optional. I assumed it wa…

It used to be that there was no official TypeScript protobuf generator from Google and third-party generators sucked. Using protobufs from web browser or in nodejs was painful.

Couple years ago Connect released very good generator for TypeScript, we use in in production and it's great:

https://github.com/connectrpc/connect-es

Re: Protobuffers Are Wrong (2018)

#158

Post by wvenable »

tgma wrote: post #155
wvenable wrote: post #134

Earlier quoted context omitted.

Except that most often you can just look at it and figure it out.

Sure you can look at it[1], but you're not expected to look at Apple Photos database. The computer is. Write a correct JSON parser, compare with protobuf on various metrics, and then we can talk. [1]: although to be fair, I am older than kids whose first programming language was JavaScript, so I do not think of JSON object format with property names in quotes and integers that need to be wrapped as strings to be safe…

I'm also "older" but I don't think that means anything.

> Sure you can look at it[1], but you're not expected to look at Apple Photos database.

How else are you supposed to figure it out? If you're older then you know that you can't rely on the existence or correctness of documentation. Being able to look at JSON and understand it as a human on the wire is huge advantage. JSON being pretty simple in structure is as advantage. I don't see a problem with quoting property names! As for large integers and datetimes, yes that could be much better designed. But that's true of every protocol and file format that has any success.

JSON parsers and writers are common and plentiful and are far less crazy than any complete XML parser/writer library.

Re: Protobuffers Are Wrong (2018)

#159

Post by spectraldrift »

I'm not sure why this post gets boosted every few years- and unfortunately (as many have pointed out) the author demonstrates here that they do not understand distributed system design, nor how to use protocol buffers. I have found them to be one of the most useful tools in modern software development when used correctly. Not only are they much faster than JSON, they prevent the inevitable redefinition of nearly identical code across a large number of repos (which is what i've seen in 95% of corporate codebases that eschew tooling such as this). Sure, there are alternatives to protocol buffers, but I have not seen them gain widespread adoption yet.

Re: Protobuffers Are Wrong (2018)

#160

Post by klodolph »

ziml77 wrote: post #153
klodolph wrote: post #96

Protobuffers suck as a core data model. My take? Use them as a serialization and interchange format, nothing more. > This puts us in the uncomfortable position of needing to choose between one of three bad alternatives: I don’t think there is a good system out there that works for both serialization and data models. I’d say it’s a mostly unsolved problem. I think I am happy with protobufs. I know that I have to fight…

> My take? Use them as a serialization and interchange format, nothing more. Isn't that exactly what they're intended for? I'm confused how anyone would even think to use them any other way.

Like the author said, their usage in practice often creeps outside that.
Post reply on HN