Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

31–40 of 307 posts

Re: Arguing against using protobuffers

#31

Post by _dmurph »

This feels pretty vitriolic, I wouldn't be surprised if there is some bias here. A lot of these problems seem pretty minor and there's weird stuff like

"Unlike most companies in the tech space, paying engineers is one of Google's smallest expenses."

According to here https://www.quora.com/How-many-software-engineers-does-Googl... there are ~28,000 engineers in 2014, which paying at 120,00 a year (glassdoor) would be... $3.3 billion a year? And this was 2014, and doesn't include stock compensation

Don't get me wrong - there are definitely problems with protobufs. But this post is citing a lot of stuff to back itself up that doesn't add up.

Also, required vs optional - definitely only optional, there should be nothing required. So I flat out disagree with that one.

Re: Arguing against using protobuffers

#32

Post by nirelBRACIT0S »

atombender wrote: post #25

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code gen…

Write your own generator then?

Re: Arguing against using protobuffers

#33

Post by romed »

It's abundantly clear why this author lasted only a year at Google. Aside from using the non-idiomatic "protobuffers" ...

"nothing wants to inspect only some bits of a message and then forward it on unchanged"

In my experience it is extremely useful to partially parse a protobuf, or to not parse it at all and simply modify it by appending to it. Also useful is the ability to define a message type that is isomorphic on the wire with a different type, but is much cheaper to parse (example: a deeply nested structure where all of the fields of interest are defined at the top level).

Nobody even within Google will argue that no mistakes have been made with proto. Proto3 is generally acknowledged to have been a big mistake and it doesn't have a lot of traction within Google. Map and OneOf are both basically antagonistic to high performance application code, and people avoid those for that reason. But the author's arguments are pretty weak and his dismissal of Google itself as an existence proof is insufficiently supported.

Re: Arguing against using protobuffers

#34

Post by sonnyblarney »

tlarkworthy wrote: post #9
sonnyblarney wrote: post #2

Good points, but can someone articulate what the best alternative to protobuffers would be in 2018, you know with 'hindsight' etc.?

JSON or graphql. Personally I think the typing, API compatibility and extensive language support make protobuffers currently the winner though, despite the warts

I like GraphQL schema, but it would have to have a standardized serialization aspect. Moreover, it really is designed for querying and has a bunch of stuff totally irrelevant to data.

But I agree ... I almost wish they'd finish it off and go for full on serialization.

That said, it may not be that-that hard, a serialization spec can be pretty short if need be. Certainly one could 'roll one's own' for a mid-sized project if need be.

Re: Arguing against using protobuffers

#35

Post by hardwaresofton »

vthallam wrote: post #22
tlarkworthy wrote: post #9

Earlier quoted context omitted.

JSON or graphql. Personally I think the typing, API compatibility and extensive language support make protobuffers currently the winner though, despite the warts

How would graphql help reduce the latency of API calls between services?

I'm no GraphQL expert/fan but one example is requesting only the data you need -- you can cut down on what you have to send over the wire which is at least guaranteed to help some.

Also, the nested query structure can save what would have normally been multiple requests with a by-the-book RESTful HATEOAS setup.

Re: Arguing against using protobuffers

#36

Post by justicezyx »

wtetzner wrote: post #29
justicezyx wrote: post #8

Wow, some minor complaints on language formality issues, then discredit the thing as a whole. But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential. It's like complaining a car that drives fast and has great gas efficiency of cannot talk. (Well in 2018, it might be less of a stretch to demand a car that can…

> But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential. Huh? It's a serialization format. This problem has been solved many times in many different ways. The author is pointing out that Protobufs were designed poorly, and they didn't have to be.

> This problem has been solved many times in many different ways.

Mind provide examples of such tools.

Re: Arguing against using protobuffers

#37

Post by vl »

_dmurph wrote: post #21
vl wrote: post #14

Earlier quoted context omitted.

If you would continue reading, there is an explanation of how optional would work almost immediately after this quote.

Right - this is the same as having the possibility of both required and optional. I'm saying there should not be any possibility of a 'required' field

In context of current protobuf design required was a mistake and optional is clearly better, but author argues about grand type system that is based on different principles, including stronger validation.

Criticizing protobufs is like criticizing C++ or Java. Both have major shortcomings, but solve practical problems and de facto lingua franca with no practical solutions to replace them.

Re: Arguing against using protobuffers

#38

Post by zamalek »

twic wrote: post #4

> Despite map fields being able to be parameterized, no user-defined types can be. This means you'll be stuck hand-rolling your own specializations of common data structures. What a pain! Well, at least Google won't make that mistake again!

My contention with the quoted text is that you probably shouldn't be using elaborate data structures in streams/files. Using DTOs as heap/stack data has bitten me enough times that I'm fairly certain that it's an anti-pattern.

It doesn't matter if you're using a quantum binomial tree in a black hole: save it as a 'stupid' map when it hits the network. That way everyone who interacts with your service can decide how they want to represent that structure. You can compose any in-memory data structure you can dream of, you can validate the data with more richness than "missing" or "present", and Protobuf doesn't contaminate your codebase.

Option 3 is the correct choice. Serialization libraries should be amateurish by design.

Re: Arguing against using protobuffers

#40

Post by docker_up »

Put me firmly in the camp of "optional fields are bad." I believe all fields should be required.

I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility.

The idea of "forwards compatibility" is a pipedream, in my opinion. Just because something won't crash because a field is or isn't there doesn't mean that it will actually work. Code ages over time, and instead of looking at the spec, which will tell you which fields are required, you need to read the documentation, which may or may not tell you what is required. And especially when dealing with an older client, it simply may not work because what used to be optional-optional fields become required-optional fields.

It's the same argument as schemaless NOSQL database like Mongo vs MySQL. Having a schema is a pain in the ass, but it's better over time and ages much more gracefully than schemaless ones, because things won't just break.

Post reply on HN