Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Protobuffers Are Wrong (2018)

reasonablypolymorphic.com

Post reply on HN
Print view View original

251–260 of 321 posts

Re: Protobuffers Are Wrong (2018)

#251

Post by ardit33 »

motorest wrote: post #217
lalaithion wrote: post #7

Protocol buffers suck but so does everything else. Name another serialization declaration format that both (a) defines which changes can be make backwards-compatibly, and (b) has a linter that enforces backwards compatible changes. Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. And I know the article says no one uses the backwards compatible…

> Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. What I dislike the most about blog posts like this is that, although the blogger is very opinionated and critical of many things, the post dates back to 2018, protobuf is still dominant, and apparently during all these years the blogger failed to put together something that they felt was a bett…

JSON exists, and when compressed it is pretty efficient. (not as efficient as protobuff though).

For client facing protocol Protobufs is a nightmare to use. For Machine to Machine services, it is ok-ish, yet personally I still don't like it.

When I was at Spotify we ditched it for client side apis (server to mobile/web), and never looked back. No one liked working with it.

Re: Protobuffers Are Wrong (2018)

#252

Post by sevensor »

vouwfietsman wrote: post #245
sevensor wrote: post #222

Earlier quoted context omitted.

Seems like a lot of effort to avoid adding a message version field. I’m not a web guy, so maybe I’m missing the point here, but I always embed a schema version field in my data.

I get that. The point is that its hard to prevent asymmetry in message versions if you are working with many communicating systems. Lets say four services inter-communicate with some protocol, it is extremely annoying to impose a deployment order where the producer of a message type is the last to upgrade the message schema, as this causes unnecessary dependencies between the release trains of these services. At the…

I think I see what you’re getting at? My mental model is client and server, but you’re implying a more complex topology where no one service is uniquely a server or a client. You’d like to insert a new version at an arbitrary position in the graph without worrying about dependencies or the operational complexity of doing a phased deployment. The result is that you try to maintain a principled, constructive ambiguity around the message schema, hence asymmetrical fields? I guess I’m still unconvinced and I may have started the argument wrong, but I can see a reasonable person doing it that way.

Re: Protobuffers Are Wrong (2018)

#253

Post by kentonv »

lostdog wrote: post #230
kentonv wrote: post #211

Earlier quoted context omitted.

Hi, I'm the apparently-FUD-spreading Cap'n Proto author. Sure! You could certainly imagine extending Protobuf or Cap'n Proto with a way to specify validation that only happens when you explicitly request it. You'd then have separate functions to parse vs. to validate a message, and then you can perform strict validation at the endpoints but skip it in middleware. This is a perfectly valid feature idea which many peop…

It gets super frustrating to have to empty/null check fields everywhere you use them, especially for fields that are effectively required for the message to make sense. A very common example I see is Vec3 (just x, y, z). In proto2 you should be checking for the presence of x,y,z every time you use them, and when you do that in math equations, the incessant existence checks completely obscure the math. Really, you wan…

> It gets super frustrating to have to empty/null check fields everywhere you use them, especially for fields that are effectively required for the message to make sense.

That's why Protobuf and Cap'n Proto have default values. You should not bother checking for presence of fields that are always supposed to be there. If the sender forgot to set a field, then they get the default value. That's their problem.

> just assuming the fields exist in code and crashing on null

There shouldn't be any nulls you can crash on. If your protobuf implementation is returning null rather than a default value, it's a bad implementation, not just frustrating to use but arguably insecure. No implementation of mine ever worked that way, for sure.

Re: Protobuffers Are Wrong (2018)

#254

Post by sevensor »

vineyardmike wrote: post #250
sevensor wrote: post #222

Earlier quoted context omitted.

Seems like a lot of effort to avoid adding a message version field. I’m not a web guy, so maybe I’m missing the point here, but I always embed a schema version field in my data.

Idk I generally think “magic numbers” are just extra effort. The main annoyance is adding if statements everywhere on version number instead of checking the data field you need being present. It also really depends on the scope of the issue. Protos really excel at “rolling” updates and continuous changes instead of fixed APIs. For example, MicroserviceA calls MicroserviceB, but the teams do deployments different time…

I think this is only a problem if you’re using a weak data interchange library that can’t use the schema number field to discriminate a union. Because you really shouldn’t have to write that if statement yourself.

Re: Protobuffers Are Wrong (2018)

#255

Post by vouwfietsman »

sevensor wrote: post #252
vouwfietsman wrote: post #245

Earlier quoted context omitted.

I get that. The point is that its hard to prevent asymmetry in message versions if you are working with many communicating systems. Lets say four services inter-communicate with some protocol, it is extremely annoying to impose a deployment order where the producer of a message type is the last to upgrade the message schema, as this causes unnecessary dependencies between the release trains of these services. At the…

I think I see what you’re getting at? My mental model is client and server, but you’re implying a more complex topology where no one service is uniquely a server or a client. You’d like to insert a new version at an arbitrary position in the graph without worrying about dependencies or the operational complexity of doing a phased deployment. The result is that you try to maintain a principled, constructive ambiguity…

Yes thats a big part, but even bigger is just the alignment of teams.

Imagine team A building feature XYZ Team B is building TUV

one of those features in each team deals with messages, the others are unrelated. At some point in time, both teams have to deploy.

If you have to sync them up just to get the protocol to work, thats an extra complexity in the already complex work of the teams.

If you can ignore this, great!

It becomes even more complex with rolling updates though: not all deployments of a service will have the new code immediately, because you want multiple to be online to scale on demand. This creates an immediate necessary ambiguity in the qeustion: "which version does this service accept?" because its not about the service anymore, but about the deployments.

Re: Protobuffers Are Wrong (2018)

#256

Post by vkou »

tyleo wrote: post #133
swiftcoder wrote: post #88

Earlier quoted context omitted.

> We use the version number to run a series of steps on each proto to upgrade old fields to new ones It sounds like you've built your own back-compat functionality on top of protobuf? The only functionality protobuf is giving you here is optional-by-default (and mandatory version numbers, but most wire formats require that)

Yeah, I’d probably say something more like, “we leverage protobuf built ins to make a slightly more advanced back compat system” We do rename deprecated fields and often give new fields their names. We rely on the field number to make that work.

> We do rename deprecated fields and often give new fields their names. We rely on the field number to make that work.

Why share names? Wouldn't it be safer to, well, not?

Re: Protobuffers Are Wrong (2018)

#257

Post by motorest »

ardit33 wrote: post #251
motorest wrote: post #217

Earlier quoted context omitted.

> Just with those two criteria you’re down to, like, six formats at most, of which Protocol Buffers is the most widely used. What I dislike the most about blog posts like this is that, although the blogger is very opinionated and critical of many things, the post dates back to 2018, protobuf is still dominant, and apparently during all these years the blogger failed to put together something that they felt was a bett…

JSON exists, and when compressed it is pretty efficient. (not as efficient as protobuff though). For client facing protocol Protobufs is a nightmare to use. For Machine to Machine services, it is ok-ish, yet personally I still don't like it. When I was at Spotify we ditched it for client side apis (server to mobile/web), and never looked back. No one liked working with it.

> JSON exists (...)

The blog post leads with the personal assertion that "ad-hoc and built by amateurs". Therefore I doubt that JSON, a data serialization language designed by trimming most of JavaScript out and to be parses with eval(), would meet the opinionated high bar.

Also, JSON is a data interchange language, and has no support for types beyond the notoriously ill-defined primitives. In contrast, protobuf is a data serialization language which supports specifying types. This means that for JSON, to start to come close to meet the requirements met by protobuf, would need to be paired with schema validation frameworks and custom configurable parsers. Which it definitely does not cover.

Re: Protobuffers Are Wrong (2018)

#259

Post by sgammon »

sgammon wrote: post #258

> Your guess is as good as mine for why an enum can’t be used as a map key. I filed an issue requesting this and it was denied with an explanation: https://github.com/protocolbuffers/protobuf/issues/7791#issu...

> It’s impossible to differentiate a field that was missing in a protobuffer from one that was assigned to the default value.

This is purportedly fixed in proto3 and latest SDK copies (IIRC)

Re: Protobuffers Are Wrong (2018)

#260

Post by sgammon »

sgammon wrote: post #258

> Your guess is as good as mine for why an enum can’t be used as a map key. I filed an issue requesting this and it was denied with an explanation: https://github.com/protocolbuffers/protobuf/issues/7791#issu...

> Contrast this behavior against message types. While scalar fields are dumb, the behavior for message fields is outright insane.

The reason messages are initialized is that you can easily set a deep property path:

```

message SomeY { string example = 1; }

message SomeX { SomeY y = 1; }

```

later, in java:

```

SomeX some = SomeX.newBuilder();

some.getY().setExample("hello"); // does not produce npe

```

in kotlin this syntax makes even more sense:

```

some {

  y.example = "hello". // does not produce npe
}

```

Post reply on HN