Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

241–250 of 307 posts

Re: Arguing against using protobuffers

#241

Post by [deleted] »

dweis wrote: post #66
docker_up wrote: post #40

Put me firmly in the camp of "optional fields are bad." I believe all fields should be required. I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility. The idea of "forwards compatibility" is a pipedream, in my opinion.…

How do you handle binary rollbacks and rollouts safely with an "everything is required" approach? Do you force binaries to roll out in a strict order with appropriate soak time at each layer? How does that affect developer velocity?

[deleted]

Re: Arguing against using protobuffers

#242

Post by [deleted] »

vkou wrote: post #209
docker_up wrote: post #193

Earlier quoted context omitted.

I doubt any api call trying to work in such a chaotic environment would actually work, and having all optional fields won't magically make things work. It will probably fail but in very mysterious ways. This sounds more like an environment where microservices are completely out of control and chaotic.

Doubt all you will, but being able to update different pieces of your service independently is a huge win. Having to update everything in lockstep is what I'd describe as 'out of control'. Which is precisely what forcing every field to be required does. There's a million and one reasons for why you may want to push, or rollback only 1 out of X services. If you follow a few simple rules when adding/removing fields, yo…

[deleted]

Re: Arguing against using protobuffers

#243

Post by sumeno »

_fq4v wrote: post #46
romed wrote: post #33

It's abundantly clear why this author lasted only a year at Google. Aside from using the non-idiomatic "protobuffers" ... "nothing wants to inspect only some bits of a message and then forward it on unchanged" In my experience it is extremely useful to partially parse a protobuf, or to not parse it at all and simply modify it by appending to it. Also useful is the ability to define a message type that is isomorphic o…

Ah right... when you can't attack the ideas, attack the author. Sandy is a Haskell enthusiast, and it is not a surprise that his criticism of protobuf is inspired heavily by that. The argument presented is that protobuf does not attempt to even replicate the best practices we already have in terms of data representation. It makes sense why languages like C are constrained in their data representations -- they want to…

> Ah right... when you can't attack the ideas, attack the author.

This feels unintentionally ironic considering the article starts with "They're clearly written by amateurs, unbelievably ad-hoc, mired in gotchas, tricky to compile, and solve a problem that nobody but Google really has"

Re: Arguing against using protobuffers

#244

Post by docker_up »

vkou wrote: post #209
docker_up wrote: post #193

Earlier quoted context omitted.

I doubt any api call trying to work in such a chaotic environment would actually work, and having all optional fields won't magically make things work. It will probably fail but in very mysterious ways. This sounds more like an environment where microservices are completely out of control and chaotic.

Doubt all you will, but being able to update different pieces of your service independently is a huge win. Having to update everything in lockstep is what I'd describe as 'out of control'. Which is precisely what forcing every field to be required does. There's a million and one reasons for why you may want to push, or rollback only 1 out of X services. If you follow a few simple rules when adding/removing fields, yo…

You don't need to update everything lockstep. You maintain compatibility by converting the structs back and forth. However, all ambiguity is gone and everything is well understood.

Re: Arguing against using protobuffers

#245

Post by docker_up »

dweis wrote: post #66
docker_up wrote: post #40

Put me firmly in the camp of "optional fields are bad." I believe all fields should be required. I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility. The idea of "forwards compatibility" is a pipedream, in my opinion.…

How do you handle binary rollbacks and rollouts safely with an "everything is required" approach? Do you force binaries to roll out in a strict order with appropriate soak time at each layer? How does that affect developer velocity?

No, you create conversion routines that convert between different versions of structs. This keeps things well understood with no ambiguity. This is very easy, we were doing this 20 years ago and autogenerating the conversions using ANTLR that parsed the XDR files for ONC/RPC.

Re: Arguing against using protobuffers

#246

Post by malkia »

AstralStorm wrote: post #205
malkia wrote: post #175

Earlier quoted context omitted.

What makes you say that Protobufs are bad thing? I'm finding the opposite - where you end up with everyone's favourite json parser, not able to handle UTF8 here and there, or some handle multi-line json, other handle comments, etc, etc. Also certainly slower, and certainly when you process data back you have to serialize, convert, check, etc. For me protobuf+grpc gives you enough building block, not to shoot yourself…

It doesn't complicate just build systems, because of the weaknesses of it's type system and code generation it requires wrappers. Those get desynced and take maintenance. This is the main point where the whole reason to use Protobuf falls apart - it doesn't work well enough to skip manual parsing. And if you do manual parsing you might well go all the way in and use JSON with a schema and a reasonable library to hand…

With certain build systems, this is a solved problem, or at least solved well, but it's a buy-in, so no great solution for MSBuild (for example), but works well in bazel.

Re: Arguing against using protobuffers

#247

Post by kentonv »

Dylan16807 wrote: post #227
kentonv wrote: post #128

Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways. This article appear…

> In every single case where you might want a repeated oneof, you always want to wrap it in a message (product type), and then repeat that. That's exactly what you can do with the existing design. But I could use the same argument for repeated anything . Why even allow repeated primitives at all, if your argument is convincing?

Repeated primitives were in the first version, before anyone knew better. I'd argue against them in new code.

In Cap'n Proto, I made it possible to upgrade a repeated primitive to a repeated strict whose first field is the primitive type, to avoid this trap.

Re: Arguing against using protobuffers

#248

Post by derefr »

fixermark wrote: post #233
derefr wrote: post #87

Earlier quoted context omitted.

> every application has to be updated when a field is added, even if they do not use that field This is when, in a protocol, you reach for the hammer called "extensibility." In this case (decoding to native structs), you'd probably have your FooMessage product-type have an "extensions" field, which is a list of zero or more FooExtensionStructs, where a FooExtensionStruct is a sum type of the known extensions to FooMe…

I believe you've also reinvented optional fields in a more generalizable way. Generalizability can be good or bad, depending on how much complexity it adds.

Nope—there's a very important difference in the two approaches. You have to choose to add the kind of extensibility I was describing to a particular type, in advance, as part of that type's original specification. You have no choice in having optional fields.

With Protobuf optional fields, even if you formally specify a protocol where some particular message is absolutely "final" and will never be extended, anyone can just throw an optional member on there when sending one to you, and your implementation won't reject the struct. This is a big problem if you're trying to specify types that really map one-to-one to fixed-size structs you'll use in the business logic of your non-memory-managed language.

With explicit extension points, you get to dictate where and when extension happens; and, as well, you get to formalize exactly what extensions look like (i.e. you don't have to allow for un-recognized extensions; you can simply have extensions be a finite set of known new extensions, where everything else is a real decode error.)

Re: Arguing against using protobuffers

#249

Post by virtualized »

pdpi wrote: post #212
_dmurph wrote: post #31

This feels pretty vitriolic, I wouldn't be surprised if there is some bias here. A lot of these problems seem pretty minor and there's weird stuff like "Unlike most companies in the tech space, paying engineers is one of Google's smallest expenses." According to here https://www.quora.com/How-many-software-engineers-does-Googl... there are ~28,000 engineers in 2014, which paying at 120,00 a year (glassdoor) would be.…

Thinking of that problem in terms of engineers not being the biggest expense is also a silly way to approach the problem. At 20000 engineers, it’s a net positive to add a full time person to save everybody else 5 seconds a day. If protobuf were such a big waste of engineering time, then working to make it more efficient for engineers while being no worse on the wire would be worth a lot of dedicated engineers.

That's why Microsoft made sure that Windows Update always runs in the background and never reboots during a presentation. Oh wait..

Re: Arguing against using protobuffers

#250

Post by repolfx »

patrec wrote: post #219

> All you've managed to do is decentralize sanity-checking logic from a well-defined boundary and push the responsibility of doing it throughout your entire codebase. One wonders if the author has practical experience with evolving a complex system composed of multiple independent services without downtime. Centralized sanity checking conflicts with transitioning services stepwise from e.g. foo.a foo.new_a by first d…

He meant "centralised in the serialisation library". I don't see how that conflicts with versioned evolution, especially because protobufs historically did verify the presence of fields.

The story behind "required considered harmful" at Google is really quite shameful. I was there at the time and couldn't quite believe people were making that argument. Beyond all the logical problems with it, they were basically saying Jeff Dean didn't know what he was doing when he made that decision (except they never actually said it explicitly because the notion itself would have been subjected to ridicule).

I used to like protobufs but I wouldn't use proto3. It seems to have regressed over time instead of getting better. Partly this is because the wire format is not evolvable. Protobufs do not have a "this is protobufs v1" header in them. This also explains the type system issues: Google cannot/will not evolve the wire format so each new version of protobufs is just different tooling and APIs over it.

Post reply on HN