Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arguing against using protobuffers

reasonablypolymorphic.com

Post reply on HN
Print view View original

221–230 of 307 posts

Re: Arguing against using protobuffers

#221

Post by jeremyjh »

docker_up wrote: post #193
taeric wrote: post #161

Earlier quoted context omitted.

You are missing when you have a middle layer. Message comes in at v3 and hits a layer that only knows v1 then gets passed to a layer that is at v4. I'd wager most places don't have that many layers. But, if you are embracing microservices, you'll find yourself here fairly fast.

I doubt any api call trying to work in such a chaotic environment would actually work, and having all optional fields won't magically make things work. It will probably fail but in very mysterious ways. This sounds more like an environment where microservices are completely out of control and chaotic.

No one said that all fields should be optional, only that it should be possible to have optional fields. It may sound like chaos to those who've not worked in such environments, but in fact it is not at all unusual in large companies to have the same message pass through several layers of services in a single call-flow. This long predates the term "microservices".

Re: Arguing against using protobuffers

#222

Post by mlthoughts2018 »

hermanradtke wrote: post #197
mlthoughts2018 wrote: post #70

Earlier quoted context omitted.

Thrift suffers pretty much all the same problems as protobufs and has many similar dysfunctional failure modes, like places that rely on serializing into Thrift structs stores in hdfs and treating that like a de facto database, with Thrift struct definitions as the schema. It is so miserable to work in code bases like that.

This is part of the original lambda architecture. What would you recommend the schema be represented in instead? Or are you preferring something like NewSQL?

I can’t tell if you mean a schema for the generated RPC or just for data storage. I’m only talking about data storage. But for RPCs, I just think don’t autogenerate them. Just expose an RPC API and let other people write code to consume it in whatever language’s packages for web requests, and don’t ever transmit things that are expected to automatically be treated as any type of object. Just send JSON or an equivalent thing in a more optimized buffer that is never allowed to be anything but a key-value store for primitive types.

On the data side, use relational database systems. Yes, even for huge modern webscale event data for apps or services with hundreds of millions of users. Don’t ever use hadoop, period.

If you’re bigger than hundreds of millions of users or otherwise are generating hundreds of billions of records per day or more, that’s the size when you might _start_ considering something different than sharded and distributed standard RDBMSs, but you are likely big enough at that point that you need an in-house, highly customized version of a distributed file store that matches your usage patterns and cost optimizations in a way that a one-size-fits-all solution cannot, and so again you should never be using hadoop.

As a result, if you find yourself relying on Thrift data serialized on a distribted filesystem as if those “flat files” are a database and map-reduce is like your de facto table scanner, it’s a red alarm bad code smell that you are growing your data scale in a horribly broken way that’s going to cause huge problems the minute you have products which need a data model or some flexibility that cannot be supported when you’ve welded data infrastructure to Thrift objects.

Re: Arguing against using protobuffers

#223

Post by lmm »

saalweachter wrote: post #120
vl wrote: post #37

Earlier quoted context omitted.

In context of current protobuf design required was a mistake and optional is clearly better, but author argues about grand type system that is based on different principles, including stronger validation. Criticizing protobufs is like criticizing C++ or Java. Both have major shortcomings, but solve practical problems and de facto lingua franca with no practical solutions to replace them.

From a practical standpoint the problem is that "required" handles a trivial subset of message validation. I mean, I'm not going to claim that it never happens that your only constraint on a valid value of a message field is "present", but you quite often want to be able to require that one of three fields is set, or a number be between 0 and 1048576, or that a field be equal to an existing user ID, it that a string…

> it turns out that "this field should exist" doesn't save that much because you still have to write "... and have a sensible value" in your own code.

Disagree. Dozens of "x should exist" checks are tedious to write and even more tedious to read, obscuring the more relevant business-specific validation logic. Better to move the low-hanging fruit into the message format.

Re: Arguing against using protobuffers

#225

Post by Dylan16807 »

the_other_guy wrote: post #178
jupp0r wrote: post #173

Earlier quoted context omitted.

Protobuf is not size efficient. The half self-describing nature and built-in backwards/forwards compatibility features sacrifice a lot of bits in efficiency. I'm not saying that this is a bad thing, just that choosing protobuf if bandwidth efficiency is your main goal is probably not a good idea.

See my other reply below under this comment tree, I also tried flatbuffers and it was extremely inefficient compared to protobuf but I don't have numbers currently for it

"msgpack and flatbuffers are worse" is faint praise.

Re: Arguing against using protobuffers

#226

Post by jamesfmilne »

marssaxman wrote: post #153
malkia wrote: post #61

I spent 2.5 years at Google, and most of what I did was pushing one protobuf from one place to another :) - and I loved it... Honestly though, you can complain all day, but some of the decisions made in the list you presented most likely come from experience (daily) not as a user of protobufs (which I simply was), but someone that had to support a plethora of compression formats, how protobufs gets stored in the diff…

Wow, different strokes indeed. Discovering that most of what I was supposed to do at google was going to consist of pushing protobufs from one place to another completely destroyed my enthusiasm for working there. I hated it!

Come and work in the film industry, where you will also discover it is still mostly plumbing, moving data from one place to another.

(Hello from another fellow Real Software alumnus :D)

Re: Arguing against using protobuffers

#227

Post by Dylan16807 »

kentonv wrote: post #128

Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways. This article appear…

> In every single case where you might want a repeated oneof, you always want to wrap it in a message (product type), and then repeat that. That's exactly what you can do with the existing design.

But I could use the same argument for repeated anything. Why even allow repeated primitives at all, if your argument is convincing?

Re: Arguing against using protobuffers

#228

Post by Dylan16807 »

amluto wrote: post #106
tathougies wrote: post #84

Earlier quoted context omitted.

Endianness issues are something every programmer should be aware of when sending data over the wire. I'm sorry I didn't insert the htons, htonls in my C code. In the Haskell code, endianness is handled by your Storable implementation (you define it after all, and can customize it however you want). I agree my example is somewhat tongue in cheek. The point though is that most languages have standard libraries for deal…

htons, etc are very 1980s, and I’ll make a fairly strong claim: they should never be used in new code, with a single exception. The reason is that an int with network endianness simply should not exist. In other words, when someone sends you a four byte big-endian integer, they sent four bytes, not an int. You can turn it into an int by shifting each byte by the relevant amount and oring them together. And a modern c…

> You can turn it into an int by shifting each byte by the relevant amount and oring them together.

Yeah! And you can even write a function to do that for you! Maybe call it "ntohl".

Re: Arguing against using protobuffers

#229

Post by seedless-sensat »

malkia wrote: post #61

I spent 2.5 years at Google, and most of what I did was pushing one protobuf from one place to another :) - and I loved it... Honestly though, you can complain all day, but some of the decisions made in the list you presented most likely come from experience (daily) not as a user of protobufs (which I simply was), but someone that had to support a plethora of compression formats, how protobufs gets stored in the diff…

Ignoring possible implementation difficulties, the application of a repeated oneof seems very esoteric. In most cases repeating the underlying fields is cleaner then repeating a oneof. E.g. instead of

    repeated oneof Option {
      Foo foo = 1;
      Bar bar = 2;
    }
use simply:

    repeated Foo foo = 1;
    repeated Bar bar = 2;
There is no longer "one of", therr are multiple, so why keep the complexity? If the interleaved ordering really matters, you can always fall back to a sub-message.

Re: Arguing against using protobuffers

#230

Post by Sir_Cmpwn »

I disagree severely with large swaths of this article. The biggest problem is here:

>Of course, the actual serialization logic is allowed to do something smarter than pushing linked-lists across the network---after all, implementations and semantics don't need to align one-to-one.

I like that I can read a pb dump in hex and have it map quite closely to the protocol definition, rather than being viewed through heaping piles of abstractions. This is only going to make it more confusing and more work to use, with the only gain being to make some functional programmer feel squishy about their beautiful type system.

I largely agree with the "The Lie of Backwards- and Forwards-Compatibility" section. You can feed /dev/urandom into nearly any protobuf definition and it'll happily give you nonsense back, no errors. I understand why this is the case, and I'm not sure it should change, but it's bitten me several times.

Post reply on HN