Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

301–307 of 307 posts

Re: Arguing against using protobuffers

#301

Post by rtfeldman »

xyzzyz wrote: post #299
rtfeldman wrote: post #296

Earlier quoted context omitted.

> Validate your data in application code, at consumption time, where you can handle errors gracefully. Honest question: how can I validate data in application code when optional fields decode to a necessarily-valid value by design? Suppose I'm an application author and I have an integer field called "quantity" which decoded to a 0. How can I tell whether that 0 meant "the quantity was 0 in the database" or "the quant…

> Suppose I'm an application author and I have an integer field called "quantity" which decoded to a 0. How can I tell whether that 0 meant "the quantity was 0 in the database" or "the quantity field was missing" instead? First, this is clear on the level of wire encoding: either the field has encoded 0 value, or it is simply missing from encoding. Second, in proto2, you actually have has_quantity() method on a proto…

Gotcha, thank you for the clear explanation!

Re: Arguing against using protobuffers

#302

Post by cromwellian »

I’ll give you one example where the protobuf design helped out a lot: GWT Protobuffers. The GWT Compiler has a version of Protobufs that transpiles all of the protos into Overlay types of naked JSON objects.

You get a statically typed Java interface, but underneath it a raw JSON object, and deserializing the proto is basically JSON.parse() + a cast.

e.g. something like this

MyProto foo = MessageDeserializer.deserialize(serializedProto); console.log(foo.getFieldFoo());

compiles to this JS:

var foo = JSON.parse(serializedProto); console.log(foo[1]);

Re: Arguing against using protobuffers

#303

Post by jiveturkey »

malkia wrote: post #61

I spent 2.5 years at Google, and most of what I did was pushing one protobuf from one place to another :) - and I loved it... Honestly though, you can complain all day, but some of the decisions made in the list you presented most likely come from experience (daily) not as a user of protobufs (which I simply was), but someone that had to support a plethora of compression formats, how protobufs gets stored in the diff…

https://memecrunch.com/meme/C2A45/protobuf-pipeline/image.jp...

Re: Arguing against using protobuffers

#304

Post by emmanueloga_ »

Agree with other ppl here, an unnecessarily violent rant, and the point the author makes is kind of lost in the title "Protobuffers Are Wrong". What they really mean is "I think Protobuffers type system could be better".

I recently saw a talk by Rich Hickey about Effective Programs [1]. The talk explains why Hickey favors dynamic types, with EDN [2] and transit [3] proposed as an alternative to more statically typed data exchange formats like protobufs and less structured ones like Json (the talk explains the reasons, nearing the end I think).

1: https://www.youtube.com/watch?v=2V1FtfBDsLU 2: https://github.com/edn-format/edn 3: https://github.com/cognitect/transit-format

Re: Arguing against using protobuffers

#305

Post by deanCommie »

PaulHoule wrote: post #235
yiyus wrote: post #217

Earlier quoted context omitted.

While I agree personal attacks do not help, he gives many reasons why he thinks Protocol Buffers are wrong. You may either respond to the issues he raises or explain what are those constraints and compromises you mention, but your comment basically just is "I'm an author, he does not know what he's talking about", which is not very productive neither.

It astonishes me how many Google products are not developer friendly. Because they think they are so freaking smart they figure they can waste their time with balky code. Protobuf raised all my red flags the first time I saw it and every time I see it again. For instance I once tested five vision recognition APIs and I could get the other ones working in 15 minutes each. The Google API went way into overtime because…

Google hires the smartest software engineers in the world.

The problem with the SMARTEST software engineers is they are incapable of distinguishing between good and bad code. They're fantastic at writing EFFICIENT code, certainly, but not code that is comprehensible by others - all code is equally comprehensible.

Am I trolling? Slightly. I am exaggerating a little bit. Obviously all engineers care about readability/maintainability to a degree. But I definitely have noted a correlation during my career between super intelligent engineers and an over-emphasis on succinctness over readability. And when provided code review feedback, they are genuinely confused "why would your version be be better? Mine is perfectly understandable"

Re: Arguing against using protobuffers

#306

Post by thegeomaster »

xyzzyz wrote: post #298
kentonv wrote: post #128

Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways. This article appear…

> OK, well, I've worked on lots of systems -- across three different companies -- where this feature is essential. Here's an actual real life example: Chrome uses this in Chrome Sync feature that allows you to sync your browser configuration and state across various devices. The feature is implemented basically like this: Chrome sends its version of state to server in a proto, and the Sync server reconciles it with t…

It's also useful the other way around - if you add a field which has significance only to the client (and these account for most fields), you don't have to count on the server knowing about this field immediately which simplifies your testing and deployment.

Re: Arguing against using protobuffers

#307

Post by kentonv »

rtfeldman wrote: post #296
kentonv wrote: post #263

Earlier quoted context omitted.

You have misunderstood the "required considered harmful" argument. It's not fundamentally about the abstract concept of required vs. optional but about the specific implementation in Protocol Buffers, which turns out to have had unintended consequences. Specifically: As implemented, required field checking occurred every time a message was serialized, not just when it was produced or consumed. Many systems involve mi…

> Validate your data in application code, at consumption time, where you can handle errors gracefully. Honest question: how can I validate data in application code when optional fields decode to a necessarily-valid value by design? Suppose I'm an application author and I have an integer field called "quantity" which decoded to a 0. How can I tell whether that 0 meant "the quantity was 0 in the database" or "the quant…

In proto2, you could use `has_foo()` to check if `foo` is present, even for integer types. You could also specify what the default value should be, so you could specify e.g. a default of -1 or some other invalid value, if zero is valid for your app.

Unfortunately, proto3 removed both of these features (`has_` and non-zero defaults). I personally think that was a mistake. I'm not sure what proto3 considers idiomatic here. Proto3 is after my time.

Cap'n Proto also doesn't support `has_` due to the nature of the encoding, but it does support defaults. So you can set a default of -1 or whatever. Alternatively, you can declare a union like:

    # (Cap'n Proto syntax)
    foo :union {
      unset @0 :Void;
      value @1 :Int32;
    }
This will take an extra 16 bits on the wire to store the tag, but gets the job done. `unset` will be the default state of the union because it has the lowest ordinal number.

I suppose in proto3, you ought to be able to use a `oneof` in a similar way.

Post reply on HN