Earlier quoted context omitted.
> This problem has been solved many times in many different ways. Mind provide examples of such tools.
Plain old JSON, with types defined.
Way too slow even at moderate scale.
Arguing against using protobuffers
171–180 of 307 posts
Re: Arguing against using protobuffers
#172Earlier quoted context omitted.
The author isn’t trying to say either of those things, really. I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, th…
No, the author is showing off how smart they are, and in an unprofessional and nasty manner to boot. It's kind of you to gift the article with some genuinely thoughtful conclusions, but you're doing all the work there, not the original. As the grandparent says, it's easy to poke holes in something, especially when disregarding important requirements that influenced its design. The fact that the OP can't point to any…
I mean, all the arguments to support my conclusions, are their arguments. If someone spends 10 pages giving you facts and anecdotes and other types of data, and then uses them to support a "dumb" conclusion, they've still done useful work. They don't realize what exactly it is their data proved, but they did do the work required to prove something. Those facts and anecdotes support a conclusion, whether or not they feed it to you at the end/in the abstract.
And, that being said, I think they're right about the conclusion, too. A shorter way to say "Protocol Buffers are a bad choice of common ESB-bus format, even though being an ESB-bus format is the primary thing they're for and what everyone tries to use them for" is "Protocol Buffers are a wrong design." If something doesn't work when used to do the thing it's advertised to do, then it's broken, even if it can do something else.
The only difference between "Protobuffers Are Wrong" and my conclusion is that I'm making the implicit context of their argument explicit. Read between the lines of their argument—they are talking about the use of protocol buffers (specifically, gRPC) in an ESB-bus common-format scenario. None of their arguments make sense if they aren't.
Re: Arguing against using protobuffers
#173Yet protobuf is probably the most compact, efficient and performant serialization method especially when saving bandwidth is important. I experimented with protofbuf, flatbuffers and messagepack and always found protobuf messages the most compact by a noticeable margin
Protobuf is not size efficient. The half self-describing nature and built-in backwards/forwards compatibility features sacrifice a lot of bits in efficiency. I'm not saying that this is a bad thing, just that choosing protobuf if bandwidth efficiency is your main goal is probably not a good idea.
Re: Arguing against using protobuffers
#174Earlier quoted context omitted.
Wow, different strokes indeed. Discovering that most of what I was supposed to do at google was going to consist of pushing protobufs from one place to another completely destroyed my enthusiasm for working there. I hated it!
I got bad news for you but almost all programming is just moving bytes from one place to another, and occasionally multiplying, adding, subtracting, or dividing some of them. ;)
I wasn't expecting really to work on a next breakthrough algorithm, and such (although folks in my team actually worked on some incredible statistics related stuff - and this is where I understood I know nothing about statistics).
Plenty for everyone, I think. I'm glad I was part of the experience!
As an ex-game tools developer (back then, and back again - current job), it was quite a change for me. If I haven't learned enough, then at least seen things from quite different perspective.
Re: Arguing against using protobuffers
#175The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…
The author isn’t trying to say either of those things, really. I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, th…
What makes you say that Protobufs are bad thing? I'm finding the opposite - where you end up with everyone's favourite json parser, not able to handle UTF8 here and there, or some handle multi-line json, other handle comments, etc, etc. Also certainly slower, and certainly when you process data back you have to serialize, convert, check, etc.
For me protobuf+grpc gives you enough building block, not to shoot yourself in the foot - but rather start from something ready. You get on top of that census metrics, bi-directional communication, and yes there are limitations - it's HTTP2 only (right? correct me if I'm wrong), but maybe that's what you need.
Instead, I see folks - going back to sock(), using them directly, and then reinventing wheels of wheels of wheels.
Then discovery comes - how do I discover what's there and there? I've got used to the nice (forgot the itnernal name) where I can ask the "grpc" server (well stubby, whatever it was called) give me your endpoints - and it gives me the endppoints. Then I can talk to it.
Yes, it complicates build systems (need to generate damn protos), but you can also do custom protobuf generator plugins that generate custom tailored access, where you get performance gains, sacrificing features you don't need, like project perfetto - https://android.googlesource.com/platform/external/perfetto/...
Here: ProtoZero - https://android.googlesource.com/platform/external/perfetto/...
Possibly old, there might be newer version, but still: https://docs.google.com/document/d/1bxlk9F79JZDk4wRXZQ9WQ5DP...
Re: Arguing against using protobuffers
#176Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways. This article appear…
[deleted]
Re: Arguing against using protobuffers
#177Hello. I didn't invent Protocol Buffers, but I did write version 2 and was responsible for open sourcing it. I believe I am the author of the "manifesto" entitled "required considered harmful" mentioned in the footnote. Note that I mostly haven't touched Protobufs since I left Google in early 2013, but I have created Cap'n Proto since then, which I imagine this guy would criticize in similar ways. This article appear…
> Maybe there's an amazing type system idea out there that would be even better, but I don't know what it is. Certainly the usual proposals I see seem like steps backwards. I'd love to be proven wrong, but not on the basis of perceived elegance and simplicity, but rather in real-world use.
Care to elaborate on what these usual proposals are and why they're backwards?
Right now I'm thinking of 'row polymorphism', which, to my understanding, is just permissiveness cooked into the type system — you get to specify types that say "I may or may not have other fields, but I definitely have a 'name' and 'email' field" for example.
Re: Arguing against using protobuffers
#178Yet protobuf is probably the most compact, efficient and performant serialization method especially when saving bandwidth is important. I experimented with protofbuf, flatbuffers and messagepack and always found protobuf messages the most compact by a noticeable margin
Protobuf is not size efficient. The half self-describing nature and built-in backwards/forwards compatibility features sacrifice a lot of bits in efficiency. I'm not saying that this is a bad thing, just that choosing protobuf if bandwidth efficiency is your main goal is probably not a good idea.
See my other reply below under this comment tree, I also tried flatbuffers and it was extremely inefficient compared to protobuf but I don't have numbers currently for it
Re: Arguing against using protobuffers
#179I like protobufs. I've only ever used them as a serialization format for save and map files for some C# games that I tinker away at, but I've been pretty happy using protobuf-net[1]. File size is smaller than JSON produced by JSON.NET or using the built-in binary serializer, much less XML. And it couldn't be much easier to use.
Re: Arguing against using protobuffers
#180The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…
If we suppose that his conclusion is using boolean logic, then what you're saying is a strawman because of his last claim; namely, protobufs are bad if "[...] && !Google": > They're clearly written by amateurs, unbelievably ad-hoc, mired in gotchas, tricky to compile, and solve a problem that nobody but Google really has. This dovetails with other arguments that I've seen recently that are becoming more frequent: Hav…
> Have we entered a new world where the lessons of companies working at massive scales are not only generally superfluous for smaller scales, but are actively harmful?
I think so, yeah.
Microservices turn out to have a lot of negative consequences, and their positives work best when you have dozens or hundreds of developers. If you've got a handful of developers... not so great.