Arguing against using protobuffers • Hacker News Board

Re: Arguing against using protobuffers

#71

Post by packetslave » Wed, Oct 10, 2018, 10:25 PM UTC

imtringued wrote: post #26

I will say one thing. gzipped JSON is damn efficient over the wire. It elegantly solves the problem that field names are repeated in every object. The only problem is that adding compression and a serializationd/deserialization step cost CPU time. However if you have this problem then you're often better off by using something like cap'n proto or flatbuffers which do not have an extra serdes step. If you consider thi…

The problem with JSON as a wire/interop format is the lack of any sort of schema. Formats like proto are nice because if you can unmarshall the bytes successfully, you can be reasonably sure you've got a valid message and reason about its contents. A deserialized JSON object can literally be or contain anything.

Re: Arguing against using protobuffers

#72

Post by cakebrewery » Wed, Oct 10, 2018, 10:28 PM UTC

justicezyx wrote: post #36

wtetzner wrote: post #29

Earlier quoted context omitted.

> But in the end, the author probably does not understand PB solves the inter-language data sharing problem, which makes all its complain secondary and inconsequential. Huh? It's a serialization format. This problem has been solved many times in many different ways. The author is pointing out that Protobufs were designed poorly, and they didn't have to be.

> This problem has been solved many times in many different ways. Mind provide examples of such tools.

Plain old JSON, with types defined.

Re: Arguing against using protobuffers

#73

Post by abernard1 » Wed, Oct 10, 2018, 10:28 PM UTC

sonnyblarney wrote: post #2

Good points, but can someone articulate what the best alternative to protobuffers would be in 2018, you know with 'hindsight' etc.?

Transit: https://github.com/cognitect/transit-format

Transit has the following features:

- self-describing

- allows extension types on top of base types

- strong value types + dynamic serialization

- can transparently pass through data without consumer knowing about schema

- has "repeated key" optimization in the encoder to allow compression in the stream

- allows transparent binary encoding with msgpack

- is stream-based, so processing can begin immediately unlike JSON which has enclosing {}

This set of features basically allow transparent proxying to JSON as well, which is something that proto2/3 cannot do without updating the proxy.

Re: Arguing against using protobuffers

#74

Post by ninjakeyboard » Wed, Oct 10, 2018, 10:31 PM UTC

I've used protocol buffers on a large project, evolving the messages over time. They are a fine choice.

The tone of this article is a little less than objective which I feel discredits it a bit. But hey it made it to the front page of hacker news so...

Re: Arguing against using protobuffers

#75

Post by atombender » Wed, Oct 10, 2018, 10:32 PM UTC

enneff wrote: post #65

atombender wrote: post #25

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code gen…

An approach I've used on the Go side is to build nice native Go types and converters to/from proto representations for wire marshaling. There's a minor performance hit but it's insignificant for what I'm doing. The result is quite nice, and you can largely ignore protobufs except for defining service APIs (which it's actually decent at). Which tool are you using for TypeScript protobuf code generation? This is the ne…

Yes, we use the same technique for our larger apps. We treat the Protobuf layer as a distinct layer related to the API; we have two packages, "protofy" and "unprotofy", so in the server implementation, you do something like:

  func (srv *Server) GetFoo(
    ctx context.Context,
    req *proto.GetFooRequest) (*proto.GetFooResponse, error) {
    foo, err := srv.db.getFoo(foo.ID)
    if err != nil { ... }
    pFoo, err := protofy.Foo(foo)
    if err != nil { ... }
    return &proto.GetFooResponse{
      Foo: pFoo,
    }, nil
  }

Obviously, a bit less pretty in reality. But the principle is the same. Having two packages makes it easier to read — e.g. protofy.Foo() is always about taking a "native" Foo and turning into a *proto.Foo, and unprotofy.Foo() is the reverse.

For TypeScript, I'm using ts-protoc-gen [1]. The weird part, which I don't fully understand, is that the serialization code is emitted as JavaScript code. All the type definitions end up in a .d.ts file, but the client is a .js file. Which means you still get type safety and autocompletion, just like plain TS, but it's still weird to me.

[1] https://github.com/improbable-eng/ts-protoc-gen

Re: Arguing against using protobuffers

#76

Post by the_clarence » Wed, Oct 10, 2018, 10:32 PM UTC

atombender wrote: post #25

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code gen…

Yup. No stream API to unmarshall protobuffers suck majorly. You have to re-invent framing

Re: Arguing against using protobuffers

#77

Post by romed » Wed, Oct 10, 2018, 10:33 PM UTC

chabsf wrote: post #41

sonnyblarney wrote: post #2

Good points, but can someone articulate what the best alternative to protobuffers would be in 2018, you know with 'hindsight' etc.?

Furthermore, in the case where I'm using protobufs as a necessary evil of grpc, what's the best alternative here?

protos are not necessary for grpc. You can use whatever payload format you want to use.

Re: Arguing against using protobuffers

#78

Post by nostrademons » Wed, Oct 10, 2018, 10:33 PM UTC

enneff wrote: post #69

weberc2 wrote: post #16

Earlier quoted context omitted.

Capnproto is built by the guy who originally built protobufs, and he cites capnproto as being better because he was able to learn from his mistakes. Makes me optimistic.

Minor clarification: the author of Capnproto worked on protobufs at Google but was not their original creator.

Minor clarification of the clarification: the author of Cap'n'proto was the original creator of Proto2, which was a ground-up (but binary compatible) rewrite of Proto1. The original authors of Proto1 were Jeff Dean & Sanjay Ghemawat. The current version of protobufs is Proto3, which AIUI is maintained by a team at Google (it was released after I left), and is an evolution of the Proto2 codebase with many new features.

Re: Arguing against using protobuffers

#79

Post by Fellshard » Wed, Oct 10, 2018, 10:34 PM UTC

hardwaresofton wrote: post #67

atombender wrote: post #25

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code gen…

I'd like to humbly suggest that we use JSON please, in particular: JSON + JSONSchema[0] +/- JSON Hyperschema[1] +/- JSON LD[2] It's a bit to learn but I promise you, it's worth it. The technologies are not redundant (jsonschema spec is for validation, hyperschema spec is for specifying how you interact, and LD is for semantics like language and more). If you take a few hours, read all 3 specs, you're almost guarantee…

I'd love to see something with the approximate structure of JSON, but less JavaScript-bound syntax. EDN has caught my eye, but then again I've always been fonder of Lisp-y syntax as a whole.

Re: Arguing against using protobuffers

#80

Post by kidsnow » Wed, Oct 10, 2018, 10:34 PM UTC

packetslave wrote: post #71

imtringued wrote: post #26

I will say one thing. gzipped JSON is damn efficient over the wire. It elegantly solves the problem that field names are repeated in every object. The only problem is that adding compression and a serializationd/deserialization step cost CPU time. However if you have this problem then you're often better off by using something like cap'n proto or flatbuffers which do not have an extra serdes step. If you consider thi…

The problem with JSON as a wire/interop format is the lack of any sort of schema. Formats like proto are nice because if you can unmarshall the bytes successfully, you can be reasonably sure you've got a valid message and reason about its contents. A deserialized JSON object can literally be or contain anything.

It could even be a gzip bomb