Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

151–160 of 307 posts

Re: Arguing against using protobuffers

#151

Post by alexashka »

dkoston wrote: post #81

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…

Maybe that's the main point, for you. I don't think I'm missing anything :)

You can take your logic and apply it to anything.

For example killing innocent babies:

"Killing babies has constraints, advantage and disadvantages. No other action in the universe is free from having advantages and disadvantages. They all have different use cases...

I don't get why the author is 'spouting dogma' about how killing innocent babies is bad.

Be wary of people who argue about killing babies etc etc."

This relativism you're expressing can be seen as ridiculous when applied to killing babies - it can actually seem to make sense when you use it to justify your own biases - which is what you're doing. Of course doing that has it's advantages and disadvantages, etc etc no opinion is better or worse everything is relative :)

Re: Arguing against using protobuffers

#152

Post by mikelward »

atombender wrote: post #25

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code gen…

Could you elaborate on what you don't like about the generated Go code?

My main complaint initially was that everything was a pointer, which made constructing things harder.

Then I realized it made zero values much easier to implement. And it lets you foo.GetA().GetB().GetC() without all the intermediate steps, which feels like a violation of the Law of Demeter, but in a glue language I think it's probably the right trade-off.

Yes, you can't use them as classes/types themselves, but you're not supposed to subclass them in any language, so they're really better thought of as structs/dicts.

Re: Arguing against using protobuffers

#153

Post by marssaxman »

malkia wrote: post #61

I spent 2.5 years at Google, and most of what I did was pushing one protobuf from one place to another :) - and I loved it... Honestly though, you can complain all day, but some of the decisions made in the list you presented most likely come from experience (daily) not as a user of protobufs (which I simply was), but someone that had to support a plethora of compression formats, how protobufs gets stored in the diff…

Wow, different strokes indeed. Discovering that most of what I was supposed to do at google was going to consist of pushing protobufs from one place to another completely destroyed my enthusiasm for working there. I hated it!

Re: Arguing against using protobuffers

#154

Post by nly »

the_other_guy wrote: post #146
nly wrote: post #134

Earlier quoted context omitted.

That really depends on the nature of your data. The space saving in protobufs really comes from its variable-length zig-zag encoding of numeric fields (including lengths for arrays and strings). Tbh, i'm surprised msgpack didn't produce smaller outputs for you, because it really should (at the cost of being slower)

No way that msgpack can produce more compact messages than protobuf, it is schemaless and stores the element names inside the message. Here is a simple benchmark I have of an array of objects where each {id: int,x: int,y: int,rot: float}. An array of 20 elements with random values of such object would take in JSON 804 bytes, in mpack 538 bytes and in protobuf3 only 304 bytes, of course this changes if you put other v…

Yep, you're right. I forgot about the keys.

Re: Arguing against using protobuffers

#156

Post by mnsc »

anothergoogler wrote: post #150
finnthehuman wrote: post #122

Earlier quoted context omitted.

>experienced engineers don’t want to work with people who think like the author of this article - has opinions on interface design - isn't afraid to be wrong publicly - is brash on a personal blog I dunno, this is mostly positive. I'd have to see how well they'd adapt to the much different context and goals of one of our design reviews, but this isn't an immediate red flag. There's a million ways to be bad at a job.…

Brash on a personal blog is tricky. In this post's case, I'd worry that the blogger has a hard time separating technical deficiencies from professional incompetence. "Designed by amateurs" is an over-the-top and dubious claim.

As an experienced engineer I can hear the pain behind that dubiuos claim and no, I wouldn't want to work with someone who behaves like this all the time but I wouldn't mind someone who once in a while gets fed up show some emotions. It's all about how you handle the aftermath, ie. can you apologize to people you accidentally hurt along the way.

Re: Arguing against using protobuffers

#157

Post by mnsc »

asdfasgasdgasdg wrote: post #125
candiodari wrote: post #119

Earlier quoted context omitted.

Nope. There are similar formats designed to be rapidly serializable and back that outperform protobufs on this front. * cap'n proto https://capnproto.org/ * flatbuffers https://google.github.io/flatbuffers/ * hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/ You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beat…

All of those significantly post-date protocol buffers, though. It would have been great if they had existed before PB, so we could have used them at Google. But, we had to invent something because nothing that existed at the time was suitable, and the migration cost to use something else now would be astonishing. I'll readily admit that a new company would likely be better off standardizing on one of these. However,…

Would you recommend someone, not Google, to standardize on protobuffers now?

Re: Arguing against using protobuffers

#158

Post by asdfasgasdgasdg »

mnsc wrote: post #157
asdfasgasdgasdg wrote: post #125

Earlier quoted context omitted.

All of those significantly post-date protocol buffers, though. It would have been great if they had existed before PB, so we could have used them at Google. But, we had to invent something because nothing that existed at the time was suitable, and the migration cost to use something else now would be astonishing. I'll readily admit that a new company would likely be better off standardizing on one of these. However,…

Would you recommend someone, not Google, to standardize on protobuffers now?

I'd probably pick Cap'n Proto or Flat Buffers if speed were paramount, in a grass is greener sort of way. I haven't used either of those technologies, though, just read about them. I'm also cool with plain JSON, which is beautiful from an ease-of-getting-started and universality perspective. I also think GraphQL is super compelling, and there's something to be said for records-as-in-SQL.

Mainly, I just think that interchange formats should be boring, simple, and ideally not incredibly slow. Protocol buffers at least meets those bars, even if it doesn't meet the consistency bar that the article desires. The whole "stop being a hipster and just use X" meme comes to mind. Probably realistically X is JSON in this day and age.

Re: Arguing against using protobuffers

#159

Post by dukoid »

dweis wrote: post #60

Hi there, I'm an actual author of Protocol Buffers :) I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises. - D P.S. I also don't beli…

Can we get access to more metadata in the lite api? Asking for a friend O:)

Re: Arguing against using protobuffers

#160

Post by bpicolo »

kentonv wrote: post #135
bpicolo wrote: post #127

Earlier quoted context omitted.

Am I missing something, or do cap'n proto and flatbuffers not support mapping types? Protobuf has disadvantages, but mapping types are something I frequently make use of. losing out on them is a big deal

Protobuf did not support maps for a long time. In practice using a list of key/value pairs works fine for most use cases. Sending the message over the wire is obviously always O(n); if you need fast inserts/lookups in code, then it usually makes sense to convert to and from the appropriate in-memory data structure for that purpose. That said, I think maps are a fine feature for a serialization format to have. But, I…

For what it's worth, I like them primarily because I don't mind using proto-generated objects as pieces passed around the stack (at least at higher levels of the stack). I lose consistency if I can do that for some objects, but am required to convert objects if they would need a mapping type.

It follows sorta the same reason that Clojure became a fairly widely used lisp – mapping types are extremely common in real world programs. Having them supported first class makes a lot of common tasks quicker to get through.

Post reply on HN