A new Go API for Protocol Buffers • Hacker News Board

Re: A new Go API for Protocol Buffers

#51

Post by lelandbatey » Tue, Mar 03, 2020, 1:46 AM UTC

atombender wrote: post #32

Hopefully this is good news. The old Protobuf package had ossified, with lots of ergonomic issues that the maintainers seemed uninterested in addressing. One perennial annoyance is how awkwardly some Protobuf stuff ends up being represented in Go. For example, "oneof" types: message Event { oneof payload { Create create = 1; Delete delete = 2; } } message Create { string id = 1; } message Delete { string version = 1;…

To quote one of the maintainers of the golang/protobuf implementation:

> The jsonpb package is intended to the be faithful implementation of the protoJSON specification, for which the C++ implementation is considered the canonical "reference" implementation.

And that C++ implementation does some kinda dumb things. For example, it marshals the protobuf type of int64 into a string when marshaling from a protobuf struct into JSON.

https://github.com/golang/protobuf/pull/916

I believe that the package they mention there is meant to be a more "canonical" protobuf json marshaller than the existing jsonpb package.

Re: A new Go API for Protocol Buffers

#52

Post by ethbro » Tue, Mar 03, 2020, 1:48 AM UTC

alkonaut wrote: post #28

dgellow wrote: post #4

> The github.com/golang/protobuf module is APIv1. > The google.golang.org/protobuf module is APIv2. We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider. That's such a weird choice, and will be quite confusing. Now every time you see protobuf being imported, you have to be sure to pay extra attention to the domain used, and correctly remember w…

Then you see the version imported being v1.X where X>20 and of course that makes it obvious that it’s v2? Are version numbers 2.0 and higher explicitly banned? Edit: found an explanation further down (spoiler: not a very good one but a very go one).

It's Google.

Valid versions are in the 0 < x < 1.0 range.

Re: A new Go API for Protocol Buffers

#53

Post by abraxas » Tue, Mar 03, 2020, 1:59 AM UTC

sigzero wrote: post #22

I thought the general consensus on protocol buffers was "Meh"?

Something like it needs to exist. JSON is stupidly inefficient for large sets of numbers. Except a framework just like grpc has been around for 25 years... but shhhh, don't tell hipsters about IIOP, they are like new parents and need to discover things for themselves.

Re: A new Go API for Protocol Buffers

#54

Post by fnord77 » Tue, Mar 03, 2020, 2:15 AM UTC

abraxas wrote: post #53

sigzero wrote: post #22

I thought the general consensus on protocol buffers was "Meh"?

Something like it needs to exist. JSON is stupidly inefficient for large sets of numbers. Except a framework just like grpc has been around for 25 years... but shhhh, don't tell hipsters about IIOP, they are like new parents and need to discover things for themselves.

do you have any links to benchmarks showing json serialization inefficiency for large sets of numbers?

I found this which shows json isn't too much worse than others, but it doesn't go into extreme cases

https://github.com/eishay/jvm-serializers/wiki

Re: A new Go API for Protocol Buffers

#55

Post by SlowRobotAhead » Tue, Mar 03, 2020, 2:18 AM UTC

xmprt wrote: post #30

myko wrote: post #29

Earlier quoted context omitted.

I think for large organizations they probably make sense. They're potentially a lot less data per request than JSON. For the average app? Overkill, sure.

I also recently learned about grpcurl that makes using protobufs a lot easier. Still not as easy as json but I think protobufs are the future.

CBOR. Binary JSON with some extra features.

I like PB, but it was too hard in my application to sync all of the sides with the same proto as it was developing rapidly.

For me schemaless was better, and now we have extremely easy JSON conversion... but did have to write the parser myself.

I personally can’t agree on PB being “the future”.

Re: A new Go API for Protocol Buffers

#56

Post by al2o3cr » Tue, Mar 03, 2020, 2:31 AM UTC

lelandbatey wrote: post #51

atombender wrote: post #32

Hopefully this is good news. The old Protobuf package had ossified, with lots of ergonomic issues that the maintainers seemed uninterested in addressing. One perennial annoyance is how awkwardly some Protobuf stuff ends up being represented in Go. For example, "oneof" types: message Event { oneof payload { Create create = 1; Delete delete = 2; } } message Create { string id = 1; } message Delete { string version = 1;…

To quote one of the maintainers of the golang/protobuf implementation: > The jsonpb package is intended to the be faithful implementation of the proto JSON specification, for which the C++ implementation is considered the canonical "reference" implementation. And that C++ implementation does some kinda dumb things. For example, it marshals the protobuf type of int64 into a string when marshaling from a protobuf struc…

IMO marshaling int64 to string is required for interoperability; RFC7159 recommends not assuming that more than IEEE754's 53 bits of integer precision are available:

https://tools.ietf.org/html/rfc7159#section-6

Re: A new Go API for Protocol Buffers

#57

Post by abraxas » Tue, Mar 03, 2020, 2:58 AM UTC

fnord77 wrote: post #54

abraxas wrote: post #53

Earlier quoted context omitted.

Something like it needs to exist. JSON is stupidly inefficient for large sets of numbers. Except a framework just like grpc has been around for 25 years... but shhhh, don't tell hipsters about IIOP, they are like new parents and need to discover things for themselves.

do you have any links to benchmarks showing json serialization inefficiency for large sets of numbers? I found this which shows json isn't too much worse than others, but it doesn't go into extreme cases https://github.com/eishay/jvm-serializers/wiki

Anything that represents numeric values as text will be an order of magnitude slower to deserialize than a binary format. I don’t have a published benchmark but in my testing which is easy to replicate I’m storing an array of around 50 million uint64 values in a contiguous block of data to maximize deserialization speed. Just to get a sense of the difference I also tested using a JSON array as a more user friendly storage format and the penalty of deserializing the JSON and populating the array was around 20x in runtime cost in my use case. The platforms I tested on were C# and Java.

Re: A new Go API for Protocol Buffers

#58

Post by kccqzy » Tue, Mar 03, 2020, 3:01 AM UTC

AaronFriel wrote: post #35

Not to be too Rust evangelist here, but why the emphasis on reflection instead of further support for generics? On the Rust side, macros can[1] transform .proto files into data structure definitions at compile time, and instances of the structs passed around have the benefit of strong typing, code completion in editors, lints and tests can easily validate properties of protobuf objects and so on. And all without addi…

That's not very scalable. Putting more logic in a macro means the compiler needs to reprocess it each time the file is touched, even when the part that's changed is not the protos. A separate code gen step means the proto compiler doesn't need to be invoked as often, and the generated files don't need to be recompiled either.

> Will anything downstream match the ".proto" of an object that's missing a bunch of required fields?

You may have missed it, but in proto3, required fields have been removed. All fields are optional now. https://stackoverflow.com/a/31814967/1102119 TL;DR: it's a code evolution problem.

Re: A new Go API for Protocol Buffers

#59

Post by randallsquared » Tue, Mar 03, 2020, 3:09 AM UTC

neild wrote: post #20

tsimionescu wrote: post #17

Earlier quoted context omitted.

> The google.golang.org/protobuf module is APIv2. We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider. This seems like a good example why tying the name of the package in the code with the place where it can be downloaded was a very bad idea. Simply changing hosting providers is now a backwards compatibility break for your clients. If Go impor…

Indeed, tying the name of a package to a hosting provider is a bad idea! That's why we changed to an import path which isn't tied to one. google.golang.org/protobuf is not tied to any particular provider, and we can redirect it wherever we want. See "go help importpath" for details on how this works.

> - Confusing: google.golang.org/protobuf@v1 doesn't exist, but v2 does.

> google.golang.org/protobuf is not tied to any particular provider, and we can redirect it wherever we want.

I agree, this is confusing. Why doesn't google.golang.org/protobuf@v1 return what's at github.com/golang/protobuf, then, if only to provide a tidy answer for those who wonder what v1 looked like?

Re: A new Go API for Protocol Buffers

#60

Post by stubish » Tue, Mar 03, 2020, 3:51 AM UTC

AaronFriel wrote: post #35

Not to be too Rust evangelist here, but why the emphasis on reflection instead of further support for generics? On the Rust side, macros can[1] transform .proto files into data structure definitions at compile time, and instances of the structs passed around have the benefit of strong typing, code completion in editors, lints and tests can easily validate properties of protobuf objects and so on. And all without addi…

A Go API for protobufs can't support generics because Go doesn't support generics, yet. I guess the protobufs team didn't want to wait until Go v2, probably a good call given Go v1.x is going to be around for years.

In Go, a code generator converts the .proto files into Go code, which give the same benefits you see with the Rust macros. Its arguable if a code generation step is better or worse than macros.

The reflection is needed if you have received a protobuf the code don't have a definition for. You can now implement something like a gRPC proxy or gRPC load balancer in Go, without needing to compile it with code from the specific .proto files. You also appear to be able to access annotations on the message definitions, which are not embedded in the generated Go structs. Rust may well have similar features in its API. Java certainly does. A gRPC proxy is a use case redact sensitive data, for when you use it to create an audit log of the messages in the requests and responses.