Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

161–170 of 307 posts

Re: Arguing against using protobuffers

#161

Post by taeric »

docker_up wrote: post #147
jeremyjh wrote: post #54

Earlier quoted context omitted.

If all fields are required, you cannot have middleware that processes multiple versions of the same protobuf, and every application has to be updated when a field is added, even if they do not use that field. This is one of the more important design goals underlying not only protobufs, but most of the non-language specific binary formatters.

> every application has to be updated when a field is added, even if they do not use that field No, you maintain the older versions of the API. V1 of the API uses the V1 struct. V2 of the API uses the V2 struct, etc. Older applications maintain compatibility because it calls the older APIs, and you can convert between V1 to V2 and only keep one version of the API. Or, if you want, you can maintain both versions of th…

You are missing when you have a middle layer. Message comes in at v3 and hits a layer that only knows v1 then gets passed to a layer that is at v4.

I'd wager most places don't have that many layers. But, if you are embracing microservices, you'll find yourself here fairly fast.

Re: Arguing against using protobuffers

#162

Post by asdfasgasdgasdg »

hardwaresofton wrote: post #126
atombender wrote: post #116

Earlier quoted context omitted.

Could one invent an ad-hoc REST API to perform joins, apply parameters and so forth? Of course. That's what people have been doing. That means every solution is ad-hoc. People get tired of this. Have you tried any of the GraphQL client tools? You can literally point a client at an arbitrary server and not only see its schema, but also interact with the API. REST doesn't give you that, because the authors of REST negl…

GraphQL certainly has a cleaner/less involved interface, but it's also less interoperable. My problem with it is that it doesn't offer much value for the amount of effort , and it isn't the paradigm shift it's claiming to be in the first place, it's just being marketed well. HateOAS[0]+Swagger[1]/jsonschema hyperschema[2] is enough to do what GraphQL does. > This is being obtuse. You know perfectly well what I meant:…

> GraphQL has offered you a way to avoid writing:

> axios.get("/posts?limit=10&filter[0]=prop&filter[1]=otherProp&fetchEmbeddedEntities[0]=thing")

Is that right? I've never used GraphQL, but I thought the main advantage was not that the client didn't have to write that, but that the server didn't have to write the handling code for the joins to the embedded entity (to pick one example), or any of the dozens of other possible joins someone might want to do for any given relation.

Re: Arguing against using protobuffers

#163

Post by asdfasgasdgasdg »

marssaxman wrote: post #153
malkia wrote: post #61

I spent 2.5 years at Google, and most of what I did was pushing one protobuf from one place to another :) - and I loved it... Honestly though, you can complain all day, but some of the decisions made in the list you presented most likely come from experience (daily) not as a user of protobufs (which I simply was), but someone that had to support a plethora of compression formats, how protobufs gets stored in the diff…

Wow, different strokes indeed. Discovering that most of what I was supposed to do at google was going to consist of pushing protobufs from one place to another completely destroyed my enthusiasm for working there. I hated it!

I got bad news for you but almost all programming is just moving bytes from one place to another, and occasionally multiplying, adding, subtracting, or dividing some of them. ;)

Re: Arguing against using protobuffers

#165

Post by hodgesrm »

dkoston wrote: post #81

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…

Protocol buffers are definitely not perfect. As of 2014 in Java the biggest issue was that you could not have different versions of protobuf [easily] running in the same VM due to the fact that protobuf generated version-dependent implementation code for IDL. So if one library depended version X of protobufs and another library depended on version Y, you had a problem because Java will only load one protobuf jar file. (Maybe it's been fixed but it was a problem then.)

That said...I used protobufs to encode the log of Tungsten Replicator, a replicator for MySQL. We did not have a single version related error while I worked on it over a period of years. That was essential given the fact that replication logs are an on-disk format that must remain stable across version upgrades. Our logs in aggregate contained billions of transactions--you can't not read an older log file just because you are running a new software version. The claim that protobufs do not handle versioning is nonsense.

Re: Arguing against using protobuffers

#166

Post by deathanatos »

zamalek wrote: post #38
twic wrote: post #4

> Despite map fields being able to be parameterized, no user-defined types can be. This means you'll be stuck hand-rolling your own specializations of common data structures. What a pain! Well, at least Google won't make that mistake again!

My contention with the quoted text is that you probably shouldn't be using elaborate data structures in streams/files. Using DTOs as heap/stack data has bitten me enough times that I'm fairly certain that it's an anti-pattern. It doesn't matter if you're using a quantum binomial tree in a black hole: save it as a 'stupid' map when it hits the network. That way everyone who interacts with your service can decide how t…

> you probably shouldn't be using elaborate data structures in streams/files.

I don't consider Option a particular "elaborate" data structure, but protobuf would benefit heavily from. Instead, as the author alludes to, you're forced to do different things depending on which side of the message/scalar dichotomy T falls on; if it's a message, you get Option essentially for free, as a submessage can always be omitted. (Which is bad in the case that you just want a T; protobufs will happily let you forget to encode that attribute, and happily decode the lack thereof at the other end, all without error, despite the message being malformed, as it lacks a way to express this — rather common — constraint.) If T is a scalar, you get to use one of the WKTs.

See this GitHub issue for more context: https://github.com/protocolbuffers/protobuf/issues/1606

I work with data where most attributes outside of the attributes that compose the key identifying some object are quite possibly unknown. Our data comes from messy datasets, and while we clean it up as best we can, sometimes an attribute's value is unintelligble in the source data, and it is more pragmatic to move on without it. In SQL, we represent this as NULL, in JSON, null, but protobuf makes it rather difficult, and this frustration is evident from the other posters in that GitHub issue.

Re: Arguing against using protobuffers

#167

Post by hodgesrm »

schizoidboy wrote: post #118
dkoston wrote: post #81

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…

If we suppose that his conclusion is using boolean logic, then what you're saying is a strawman because of his last claim; namely, protobufs are bad if "[...] && !Google": > They're clearly written by amateurs, unbelievably ad-hoc, mired in gotchas, tricky to compile, and solve a problem that nobody but Google really has. This dovetails with other arguments that I've seen recently that are becoming more frequent: Hav…

No. But you have to examine use cases carefully including the assumptions. I think this has always been the case but people (in my experience at least) get a little dazzled by the massive scale of companies like Facebook and try to apply their solutions to problems for which they are simply not applicable.

Re: Arguing against using protobuffers

#168

Post by angry_octet »

hodgesrm wrote: post #165
dkoston wrote: post #81

The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…

Protocol buffers are definitely not perfect. As of 2014 in Java the biggest issue was that you could not have different versions of protobuf [easily] running in the same VM due to the fact that protobuf generated version-dependent implementation code for IDL. So if one library depended version X of protobufs and another library depended on version Y, you had a problem because Java will only load one protobuf jar file…

Is that a problem with ProtoBufs or Java?

Re: Arguing against using protobuffers

#169

Post by hodgesrm »

angry_octet wrote: post #168
hodgesrm wrote: post #165

Earlier quoted context omitted.

Protocol buffers are definitely not perfect. As of 2014 in Java the biggest issue was that you could not have different versions of protobuf [easily] running in the same VM due to the fact that protobuf generated version-dependent implementation code for IDL. So if one library depended version X of protobufs and another library depended on version Y, you had a problem because Java will only load one protobuf jar file…

Is that a problem with ProtoBufs or Java?

Java only allows one version of any class within a single class loader. That's just how it works. So the problem was at the very least a limitation of the protobufs Java implementation.

Speaking of which, the generated code was incredibly convoluted with what seemed like tangled dependencies on the underlying protobufs library. I used to dread debugging it--sometimes I would have to go into it to figure out what the upper layers were doing wrong. Luckily it did not happen very often. And I don't recall ever hitting a bug in protobufs itself for all that the code was virtually unreadable.

Re: Arguing against using protobuffers

#170

Post by tgsovlerkhgsel »

> Make all fields in a message required. This makes messages product types.

Now you lost one of the biggest benefits of using protobufs, the possibility to make things backwards/forwards compatible. When you add a field to the proto, a server using the new version of the proto will fail to deserialize a request from a client using an older version that's missing that field.

The author mentions this later, but doesn't really address it/propose a better solution, as far as I can see.

Post reply on HN