Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

231–240 of 307 posts

Re: Arguing against using protobuffers

#231

Post by fixermark »

atombender wrote: post #75
enneff wrote: post #65

Earlier quoted context omitted.

An approach I've used on the Go side is to build nice native Go types and converters to/from proto representations for wire marshaling. There's a minor performance hit but it's insignificant for what I'm doing. The result is quite nice, and you can largely ignore protobufs except for defining service APIs (which it's actually decent at). Which tool are you using for TypeScript protobuf code generation? This is the ne…

Yes, we use the same technique for our larger apps. We treat the Protobuf layer as a distinct layer related to the API; we have two packages, "protofy" and "unprotofy", so in the server implementation, you do something like: func (srv *Server) GetFoo( ctx context.Context, req *proto.GetFooRequest) (*proto.GetFooResponse, error) { foo, err := srv.db.getFoo(foo.ID) if err != nil { ... } pFoo, err := protofy.Foo(foo) if…

I believe that means you're more or less using it as intended. Protobuffers are intended for serialization. In go, serialization is often handled at a separate layer from the business logic.

Re: Arguing against using protobuffers

#232

Post by fixermark »

docker_up wrote: post #40

Put me firmly in the camp of "optional fields are bad." I believe all fields should be required. I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility. The idea of "forwards compatibility" is a pipedream, in my opinion.…

If all fields are required, how do you add a new required field without breaking all of the clients that were using the previous protobuffer version?

Re: Arguing against using protobuffers

#233

Post by fixermark »

derefr wrote: post #87
jeremyjh wrote: post #54

Earlier quoted context omitted.

If all fields are required, you cannot have middleware that processes multiple versions of the same protobuf, and every application has to be updated when a field is added, even if they do not use that field. This is one of the more important design goals underlying not only protobufs, but most of the non-language specific binary formatters.

> every application has to be updated when a field is added, even if they do not use that field This is when, in a protocol, you reach for the hammer called "extensibility." In this case (decoding to native structs), you'd probably have your FooMessage product-type have an "extensions" field, which is a list of zero or more FooExtensionStructs, where a FooExtensionStruct is a sum type of the known extensions to FooMe…

I believe you've also reinvented optional fields in a more generalizable way. Generalizability can be good or bad, depending on how much complexity it adds.

Re: Arguing against using protobuffers

#234

Post by fixermark »

Dylan16807 wrote: post #227
kentonv wrote: post #128

Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways. This article appear…

> In every single case where you might want a repeated oneof, you always want to wrap it in a message (product type), and then repeat that. That's exactly what you can do with the existing design. But I could use the same argument for repeated anything . Why even allow repeated primitives at all, if your argument is convincing?

There is a protobuffer best practice that suggests that if you think you might need a repeated message in the future, you should use a repeated message instead of a repeated primitive conservatively.

Re: Arguing against using protobuffers

#235

Post by PaulHoule »

yiyus wrote: post #217
dweis wrote: post #60

Hi there, I'm an actual author of Protocol Buffers :) I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises. - D P.S. I also don't beli…

While I agree personal attacks do not help, he gives many reasons why he thinks Protocol Buffers are wrong. You may either respond to the issues he raises or explain what are those constraints and compromises you mention, but your comment basically just is "I'm an author, he does not know what he's talking about", which is not very productive neither.

It astonishes me how many Google products are not developer friendly. Because they think they are so freaking smart they figure they can waste their time with balky code.

Protobuf raised all my red flags the first time I saw it and every time I see it again.

For instance I once tested five vision recognition APIs and I could get the other ones working in 15 minutes each. The Google API went way into overtime because Google's libraries trashed my Python installation forcing me to reinstall.

Google made a real boner with namespace packages in Python and they've contributed a big chunk of entropy to the Java ecosystem with the Guava library that, to this day, holds back Hadoop and all of the code around it to version 13 point something because what was supposed to be a minor revision broke HDFS.

Re: Arguing against using protobuffers

#236

Post by lemmsjid »

kentonv wrote: post #128

Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways. This article appear…

Indeed, I was waiting for the punchline where he reveals a superior alternative (given the fundamental nature of his criticism it needed to be a fundamentally superior concept, to justify the rhetoric, so something that would give you an a-ha moment, rather than iterative improvements). When you open sourced protobuf (thank you for doing so), we at my existing company had implemented a functionally similar but not language agnostic implementation (as I believe many companies had), and it was very exciting to have a cross language implementation. Also to your last point one of the major drivers was the forward-backward compatability and message preservation across intermediate systems that only required knowledge of their portion of the message, so the author's assertion that this never comes up also surprised me.

Of course I am all ears for the next revolution in high performance message formats that support schema evolution, etc., but the author did not provide one, and I think if one is going to unleash that level of negative rhetoric, one takes on the responsibility to also unveil at least a glimpse at an alternative.

Re: Arguing against using protobuffers

#237

Post by TeMPOraL »

> Protobuffers correspond to the data you want to send over the wire, which is often related but not identical to the actual data the application would like to work with.(...)

> Option 1 is clearly the "right" solution, but its untenable with protobuffers. The language isn't powerful enough to encode types that can perform double-duty as both wire and application formats. Which means you'd need to write a completely separate datatype, evolve it synchronously with the protobuffer, and explicitly write serialization code between the two.

I'd say, get used to this. This is exactly the same story you have with ORM/ActiveRecord. You either treat it as a different layer and write the translation code that's needed, or mix it with internal logic and pay the price of messy codebase later. The problem isn't protobuff(or ORM)-specific. It's just a consequence of the fact that your internal business model and the data you want to save/send are usually two different things.

Re: Arguing against using protobuffers

#238

Post by fsaintjacques »

stouset wrote: post #183
asdfasgasdgasdg wrote: post #158

Earlier quoted context omitted.

I'd probably pick Cap'n Proto or Flat Buffers if speed were paramount, in a grass is greener sort of way. I haven't used either of those technologies, though, just read about them. I'm also cool with plain JSON, which is beautiful from an ease-of-getting-started and universality perspective. I also think GraphQL is super compelling, and there's something to be said for records-as-in-SQL. Mainly, I just think that int…

Absolutely no to JSON, IMO. The problem with JSON for internal formats is that there’s often only one consumer and producer, so documenting the format rarely happens. Later, when you want to reimplement one side, you learn that there is no “one place” where you parse the JSON, but that you hand bits and pieces of it to completely unrelated areas of code. Figuring out these as hoc formats is nigh impossible, so you en…

This.

Don't forget integer type defined by implementation. Very easy to shoot yourself in the foot in when consumers/producers are not written in the same language, even more if it's a mix of dynamic and static languages.

Nothing beats a well defined and commented schema.

Re: Arguing against using protobuffers

#239

Post by shabbyrobe »

mikelward wrote: post #152
atombender wrote: post #25

Though I dislike the hyperbolic tone and personal attacks, the author isn't entirely wrong. There are many design choices in Protocol Buffers that seem directly related to the scale and complexity at which Google operates, and which sacrifice safety, clarity and language integration. The utter awkwardness of Protobuf-generated code is particularly problematic. I've had pretty good results with the TypeScript code gen…

Could you elaborate on what you don't like about the generated Go code? My main complaint initially was that everything was a pointer, which made constructing things harder. Then I realized it made zero values much easier to implement. And it lets you foo.GetA().GetB().GetC() without all the intermediate steps, which feels like a violation of the Law of Demeter, but in a glue language I think it's probably the right…

Oneofs, at least with gogoproto, are quite unpleasant to work with. There's a separate intermediate type generated for each member of a oneof field. With the following message:

    message Foo {
        oneof bar {
            Baz baz = 1;
            Qux qux = 2;
        }
    }
You'll have the types Foo, Baz and Qux like you'd expect, but you also get Foo_Baz, which does nothing but wrap the Baz for use with the Foo.bar property, and Foo_Qux, which does the same thing but for Qux instead of Baz. To make matters worse, the types all implement an interface which is declared but not exported. Stringing the whole thing together to assign to a oneof looks like this:

    foo := &rpc.Foo{Bar: &rpc.Foo_Baz{Baz: &rpc.Baz{}}}
That example makes it look better than it really is; in practice, you'll probably have more properties and longer names.

Re: Arguing against using protobuffers

#240

Post by slavik81 »

jeremyjh wrote: post #221
docker_up wrote: post #193

Earlier quoted context omitted.

I doubt any api call trying to work in such a chaotic environment would actually work, and having all optional fields won't magically make things work. It will probably fail but in very mysterious ways. This sounds more like an environment where microservices are completely out of control and chaotic.

No one said that all fields should be optional, only that it should be possible to have optional fields. It may sound like chaos to those who've not worked in such environments, but in fact it is not at all unusual in large companies to have the same message pass through several layers of services in a single call-flow. This long predates the term "microservices".

> No one said that all fields should be optional

As of protobuf v3, all fields must be optional. https://github.com/protocolbuffers/protobuf/issues/2497

Post reply on HN