Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

271–280 of 307 posts

Re: Arguing against using protobuffers

#271

Post by ionforce »

angry_octet wrote: post #168
hodgesrm wrote: post #165

Earlier quoted context omitted.

Protocol buffers are definitely not perfect. As of 2014 in Java the biggest issue was that you could not have different versions of protobuf [easily] running in the same VM due to the fact that protobuf generated version-dependent implementation code for IDL. So if one library depended version X of protobufs and another library depended on version Y, you had a problem because Java will only load one protobuf jar file…

Is that a problem with ProtoBufs or Java?

I'm an outsider, but maybe it's a problem of their interaction?

Most likely, the vast majority of Protobuf use is with code generated to a single, canonical location (which leads to the problem described above).

But hypothetically, it would be possible for code to generate and refer to two different versions. That doesn't sound impossible. Maybe just improbably given the way things are typically set up.

Re: Arguing against using protobuffers

#272

Post by ionforce »

One of the most surprising features of Proto mentioned in this thread (at least for me/my background) is the desire to forward on a message you receive. Possibly a message you can't read/parse, or don't want to. And you want to forward that whole payload to the next person.

What kind of systems have this architecture? Why would you forward on a message and not know its contents 100%?

Re: Arguing against using protobuffers

#273

Post by ionforce »

tehlike wrote: post #164

I never understood debate over technology that just works You may not prefer it, but it works. Do something better, and make people use it.

The disagreement is over what "just works" means. Obviously the author things Protobuf isn't suitable for some use cases. They don't work in his view.

Re: Arguing against using protobuffers

#274

Post by ionforce »

moron4hire wrote: post #99

There are enough voices on the net these days that we don't have to put up with this kind of attitude anymore. This sort of stuff was "just the way things were" when we were teenagers visiting forums ran by teenagers. I don't know if the guy is right or wrong, I've just got better things to do than to sift through all the vitriol to get to whatever point he is trying to make.

I think a lot of people are still attracted to the "hostile genius" persona, or whatever. Maybe it justifies their own poor behavior.

Re: Arguing against using protobuffers

#275

Post by marssaxman »

jamesfmilne wrote: post #226
marssaxman wrote: post #153

Earlier quoted context omitted.

Wow, different strokes indeed. Discovering that most of what I was supposed to do at google was going to consist of pushing protobufs from one place to another completely destroyed my enthusiasm for working there. I hated it!

Come and work in the film industry, where you will also discover it is still mostly plumbing, moving data from one place to another. (Hello from another fellow Real Software alumnus :D)

Hi! That's a name I haven't seen in a while. :-)

Oddly enough, I worked on film-industry data plumbing for a couple of years before I joined Real - a distributed, fibre-channel based file system called Centravision. Just looked it up and it's apparently still in use, two acquisitions later, under the name "StorNext"...

Re: Arguing against using protobuffers

#276

Post by joshuamorton »

docker_up wrote: post #270
joshuamorton wrote: post #257

Earlier quoted context omitted.

This doesn't make sense with the concept of rollbacks. If I rollout server version 2 and client version 2 which each use a new required field, and then realize that there is some terrible error in server version 2, I can't roll it back to version 1, since it will reject all client calls from v2 clients. The only way to make it work is to add a translation layer, as you suggest, on the server, wait a while, push the n…

Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. Or you could have client version 2 know how to automatically convert to server version 1, because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1. You can't do this with protobufs because…

>Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then.

This depends on the update. If indeed the field was optional, you won't. A common example would be a field that is necessary for a new feature, but without which everything functions just fine, or functions with a minor degredation in experience.

But more importantly, you want to decouple features from api versions and communication protocols. It should be possible to enable an feature without rolling out a new version (for example, via a flipped flag). So a degraded communication protocol shouldn't actually impact anything. The application worked fine an hour ago, adding a new field won't break it.

>You have to roll back the clients as well then.

And if the clients are phones?

>Or you could have client version 2 know how to automatically convert to server version 1

This requires communicating with the server beforehand. Why should I have to roundtrip to negotiate which api version I should use? And what happens if you want to modify the protocol that you use to decide on api versions? Its turtles all the way down.

>because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1.

So now, before I can roll out a server update, I need to roll out a client update that can make sure to negotiate back to a degraded experience until I update the server. Then I have to wait until that new client is rollback safe. Then I can update the server, then eventually I can update the client to remove the shim code. That's the "strict ordering with appropriate soak time" issue that you're still running into.

This has cascading consequences, each server/client update dance has to be mostly atomic, so you can only really do one of these dances at a time, and all clients have to be in sync. If a deep dependency service wants to make an api change, it has to wait until all of the clients are prepared before updating, and if any of those clients is the server in another context, they have to wait until everyone is ready.

That's chaos. And I don't want to do that when the other option is "update the server whenever, as clients upgrade, they'll see the improved service". You avoid the dance. It moves the initiative of an upgrade from the client to the server, and this is a good thing, because there are more clients than servers.

Re: Arguing against using protobuffers

#277

Post by docker_up »

joshuamorton wrote: post #276
docker_up wrote: post #270

Earlier quoted context omitted.

Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. Or you could have client version 2 know how to automatically convert to server version 1, because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1. You can't do this with protobufs because…

>Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. This depends on the update. If indeed the field was optional, you won't. A common example would be a field that is necessary for a new feature, but without which everything functions just fine, or functions with a minor degredation in experience. But more imp…

We agree to disagree. I don't think you can convince me that all optional is better than all required and vice versa, which is okay. My point is required fields makes software age better over the long run because everything is explicit. If you don't agree, that's your prerogative. Everyone thought NOSQL without schemas was a godsend, until their code/service iterated a dozen times, developers leave, documentation gets out of date, and now all their older data can't be read because it doesn't match the code. Same thing holds true for RPC, in my opinion. Yours may differ.

Re: Arguing against using protobuffers

#278

Post by sangnoir »

kentonv wrote: post #148
oh_sigh wrote: post #129

Earlier quoted context omitted.

I thought Kenton Varda was the author of protobufs?

Nope, I'm not the original author -- that would be Jeff Dean and Sanjay Ghemawat (also often credited with inventing things like MapReduce, BigTable, Spanner, ...). I wrote version 2 (a complete rewrite, but largely following the original design) and open sourced it. I stopped working on Protobuf about 8 years ago. Many others who have been on the Protobuf team since can certainly call themselves "authors".

> that would be Jeff Dean and Sanjay Ghemawat

or "amateurs", as the post would call them.

Re: Arguing against using protobuffers

#279

Post by fixermark »

derefr wrote: post #248
fixermark wrote: post #233

Earlier quoted context omitted.

I believe you've also reinvented optional fields in a more generalizable way. Generalizability can be good or bad, depending on how much complexity it adds.

Nope—there's a very important difference in the two approaches. You have to choose to add the kind of extensibility I was describing to a particular type, in advance, as part of that type's original specification. You have no choice in having optional fields. With Protobuf optional fields, even if you formally specify a protocol where some particular message is absolutely "final" and will never be extended, anyone ca…

... however, it's a great feature if the use case is "protocol for client-server network communication," which was the design goal for protobuffers. It follows the "permissive in what you accept, strict in what you emit" design philosophy.

Re: Arguing against using protobuffers

#280

Post by killerstorm »

ionforce wrote: post #272

One of the most surprising features of Proto mentioned in this thread (at least for me/my background) is the desire to forward on a message you receive. Possibly a message you can't read/parse, or don't want to. And you want to forward that whole payload to the next person. What kind of systems have this architecture? Why would you forward on a message and not know its contents 100%?

> What kind of systems have this architecture?

The Web, for example.

> Why would you forward on a message and not know its contents 100%?

In the web architecture you can observe following cases:

* proxying, which itself is an umbrella term for a lot of different things, such as:

    * request routing

    * filtering

    * access control

    * logging

 * caching

 * load balancing
For example, a very typical scenario for load balancing is "sticky sessions". Basically, a reverse proxy (e.g. nginx) needs to inspect a HTTP request and analyze certain headers to find the server to forward request to (which already has user's session). Note that nginx can just pass headers without understanding what they mean. And it simply cannot understand the meaning of query part of URL or message body.

So, apparently, protobuf gives you a lot of HTTP's flexibility in a binary RPC format. So e.g. you can make a cache server which will cache responses without understanding them, so you can evolve your backend without updating cache server each time you add a new field to a response.

What I don't understand, however, is why they couldn't control that behavior using compiler options. Sometimes you want to be lax with validation, sometimes you don't. Why not give programmer a choice...

Post reply on HN