Live data from Hacker News

A new Go API for Protocol Buffers

blog.golang.org

31–40 of 100 posts

Re: A new Go API for Protocol Buffers

#31
post #9
post #7

Earlier quoted context omitted.

Because they would had to add a `/v2` suffix to the import path and seems like they didn't wanted to add that. The question is, why is this not released as v1.0.0 then? Still trying to understand the reasoning for this.

Reasoning went something like this: We could tag the new API v2: + Makes the "v1" and "v2" distinction very clear in the import path. - Confusing: google.golang.org/protobuf@v1 doesn't exist, but v2 does. - In ten years, hopefully nobody cares about the old github.com/golang/protobuf and the confusion is gone. We could tag the new API v1: - Less visually distinct in the import path. + Seems to make sense for the firs…

When you were waffling back on forth on the decision, did anyone note that “re-using v1 is a weird choice and is going to be the main thing people talk about”?

Re: A new Go API for Protocol Buffers

#32
Hopefully this is good news. The old Protobuf package had ossified, with lots of ergonomic issues that the maintainers seemed uninterested in addressing.

One perennial annoyance is how awkwardly some Protobuf stuff ends up being represented in Go. For example, "oneof" types:

  message Event {
    oneof payload {
      Create create = 1;
      Delete delete = 2;
    }
  }

  message Create {
    string id = 1;
  }

  message Delete {
    string version = 1;
  }
You get these messy types (I've elided most of the yucky marshaling-related stuff):

  type Event struct {
    Payload isEvent_Payload `protobuf_oneof:"item"`
  }

  type isEvent_Payload interface {
    isEvent_Payload()
    MarshalTo([]byte) (int, error)
    Size() int
  }

  type Event_Create struct {
    Create *Create `protobuf:"bytes,1,opt,name=create,proto3,oneof"`
  }

  type Event_Delete struct {
    Delete *Delete `protobuf:"bytes,1,opt,name=delete,proto3,oneof"`
  }
There are multiple problems here. One: Why the extra struct? For example, to create a single event, you have to do:

  Event{
    Payload: &Event_Create{
      Create: &Create{ID: "123"},
    },
  },
...instead of just:

  Event{
    Payload: &Create{ID: "123"},
  },
Secondly, isEvent_Payload is private! So given a Create or an Delete, you can't store it in a single variable:

  func nextEvent() isEvent_Payload { // 
The reason for this appears to be so that the generated serialization can be very dumb and not rely on type switches. But I don't buy that this is how it has to be.

There are other issues. Overall, the whole package seems designed from the bottom up for machines (mumbles possibly also by machines), not humans.The upshot is that using Protobuf types as "first-class" types — meaning the types you actually use internally in the meat of your app, as opposed to in the controller glue that lives in your API and mediates between the API and the internals — feels super messy.

As an aside, anyone know what issues this paragraph refers to?

  The google.golang.org/protobuf/encoding/protojson package
  converts protocol buffer messages to and from JSON using the
  canonical JSON mapping, and fixes a number of issues with the
  old jsonpb package that were difficult to change without
  causing problems for existing users.
Did they finally fix the zero value problem with jsonpb.go [1]?

[1] https://github.com/gogo/protobuf/issues/218

Re: A new Go API for Protocol Buffers

#33
post #29
post #22

I thought the general consensus on protocol buffers was "Meh"?

I think for large organizations they probably make sense. They're potentially a lot less data per request than JSON. For the average app? Overkill, sure.

Overkill? It doesn’t take a lot of work to add proto buffers to an existing project (if you’re already using a reasonable build system, anyway).

Re: A new Go API for Protocol Buffers

#34
post #9
post #7

Earlier quoted context omitted.

Because they would had to add a `/v2` suffix to the import path and seems like they didn't wanted to add that. The question is, why is this not released as v1.0.0 then? Still trying to understand the reasoning for this.

Reasoning went something like this: We could tag the new API v2: + Makes the "v1" and "v2" distinction very clear in the import path. - Confusing: google.golang.org/protobuf@v1 doesn't exist, but v2 does. - In ten years, hopefully nobody cares about the old github.com/golang/protobuf and the confusion is gone. We could tag the new API v1: - Less visually distinct in the import path. + Seems to make sense for the firs…

Can you explain why it's not possible to create `google.golang.org/protobuf@v1` retroactively?

Also:

> In ten years, hopefully nobody cares about the old github.com/golang/protobuf and the confusion is gone.

is the corollary that this change will cause 10 years of confusion?

Re: A new Go API for Protocol Buffers

#35
Not to be too Rust evangelist here, but why the emphasis on reflection instead of further support for generics?

On the Rust side, macros can[1] transform .proto files into data structure definitions at compile time, and instances of the structs passed around have the benefit of strong typing, code completion in editors, lints and tests can easily validate properties of protobuf objects and so on. And all without adding more codegen steps which is nice - the macros can replace a build system having to do codegen as a separate step.

How often do you need to mutate any protocol buffer and remove (redact as they say) fields generating a completely invalid protocol buffer object? Will anything downstream match the ".proto" of an object that's missing a bunch of required fields? Wouldn't it be easier to just all of the sensitive data into an optional "sensitive_data" field that's defined by another .proto, and strip that?

[1] - https://github.com/danburkert/prost#generated-code-example

Re: A new Go API for Protocol Buffers

#36
post #29

Earlier quoted context omitted.

I think for large organizations they probably make sense. They're potentially a lot less data per request than JSON. For the average app? Overkill, sure.

Overkill? It doesn’t take a lot of work to add proto buffers to an existing project (if you’re already using a reasonable build system, anyway).

If you have weeks to spend optimizing build systems, linters/editors, CI/CD, etc...

Re: A new Go API for Protocol Buffers

#37

Not to be too Rust evangelist here, but why the emphasis on reflection instead of further support for generics? On the Rust side, macros can[1] transform .proto files into data structure definitions at compile time, and instances of the structs passed around have the benefit of strong typing, code completion in editors, lints and tests can easily validate properties of protobuf objects and so on. And all without addi…

Any talk of required fields is a non-starter. Just forget about it. Don't use them.

Whether a field is "sensitive" may change after the field is defined and used all over the place, so you can't just move it without changing all the code. But you can easily add a option (like [non_sensitive=true]) to any field so your redaction code picks it up.

Re: A new Go API for Protocol Buffers

#38
post #4

> The github.com/golang/protobuf module is APIv1. > The google.golang.org/protobuf module is APIv2. We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider. That's such a weird choice, and will be quite confusing. Now every time you see protobuf being imported, you have to be sure to pay extra attention to the domain used, and correctly remember w…

The goal of issuing a major version is to create a separate package namespace so it can be imported concurrently to other major versions. By treating major versions as different packages, it creates efficiencies in VCS history and common namespace. However, the goal is to create a new package name that can be concurrently imported with previous versions. Everything else is just method. Don't confuse the goal with the…

People care about versions. If it’s harder or more confusing to understand what version you’re on, that’s more than “just method”. Otherwise we’d just use UUIDs for package names and call it a day.

Re: A new Go API for Protocol Buffers

#39
post #21

Earlier quoted context omitted.

What are you going to do for APIv3? Version it as google.golang.org@v2? google.golang.org@v3 (but @v2 doesn't exist)? Release an @v2 which is identical to @v1?

We used up our breaking change budget for the next decade with this release, so we'll think about it in 2030.

[deleted]

Re: A new Go API for Protocol Buffers

#40
post #10
post #4

> The github.com/golang/protobuf module is APIv1. > The google.golang.org/protobuf module is APIv2. We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider. That's such a weird choice, and will be quite confusing. Now every time you see protobuf being imported, you have to be sure to pay extra attention to the domain used, and correctly remember w…

> If only go modules established a clear way to differentiate API versions using the import path! It did, they just chose not to follow it :/ I wonder what rsc thinks about this decisions

Rsc?
Post reply on HN