What I was reading? It is easy to badmouth someone without knowing the context. It is not uncommon to end up with a flawed design if you find yourself in circumstances like short deadline, uncertain requirements, lack of resources, small user base, no QA and so on that are outside of the developer means to change.
Arguing against using protobuffers
141–150 of 307 posts
Re: Arguing against using protobuffers
#142Once I was faced with a task of serializing object hierarchy to a binary on disk[1], with the requirement of transparent backward compatibility (and desired forward compatibility), and the ability to be accessed from several programming languages in the future.
Protobufs looked promising because of their forward compatibility promise, but it looked like they only work if you write your code from the get-go using them. Rewiring a 15-year-old codebase would be a nightmare, so that was not an option.
Boost::serialization[2] looked promising, because it's non-intrusive -- but even "dumb" forward compatibility (skipping unknown fields) is not supported -- and reading its archives from other languages seems nontrivial.
So I went ahead and rolled out a custom framework with C++ template trickery which was non-intrusive, wrote to SQLite (with a fixed ORM schema), and did the job.
I wondered for a while if I was re-inventing the wheel, but in the end, I am glad I did it:
1. It wasn't that much work - especially compared to rewiring everything for Protobufs
2. It made me appreciate the goals and the API of Boost::serialization better - mine ended up being not too dissimilar. It was a great way to learn that - or any other- serialization API - without even looking at it! Now it's "let's see how they did ___" instead of "what does it do and why is this here".
Reading this article made me glad again: partly, for not depending on Protobuf -- and partly, for it having some of the same design flaws that I have (like having difficulty telling a zero-initialized value from a value not present in the data).
But the take-away for me was that sometimes, writing a custom framework is justified. Not all tools are suited for all purposes. Figuring out what one wants, and checking the list against the goals of a project can save a lot of headache.
[1] Binary was a necessity because some of the data is large arrays of full-precision floating point numbers, on the scale of 10-100MB. This was scientific data, no precision loss on saving/loading was acceptable.
[2]https://www.boost.org/doc/libs/1_67_0/libs/serialization/doc...
Re: Arguing against using protobuffers
#143Earlier quoted context omitted.
You're absolutely right that it is a sticking point, but custom types can help this. Also, I don't know about you but I don't very often have Infinity or NaN as inputs that I want to see in my schemas... Also, the JSON schema is an evolving document being developed in the open, it can be improved over time (they're already on Draft 7) -- if numeric types are lacking then suggest a way to make them better. > And most…
I've wanted NaN and Infinity on a few serialization worlds. Mainly because I didn't want to have to reinvent what the IEEE numbers already have. My complaint on the items being less than compelling is that they almost instantly devolve into bitter fights about what I'm supposed to care about. I fully grant that XML was too heavy in much of its schema attempts. However, it is much easier to reason about the extremity…
> My complaint on the items being less than compelling is that they almost instantly devolve into bitter fights about what I'm supposed to care about. I fully grant that XML was too heavy in much of its schema attempts. However, it is much easier to reason about the extremity of XML than it is where in the middle road I want to be in.
Could you give a more concrete example? I'm really not following -- JSON + jsonschema is almost equivalent to XML + XSD in my mind, what was it that XML/XSD gave you that didn't devolve into bitter fights?
> That it is an evolving document being developed in the open is worthless if it is not always maintaining backwards compatibility. And, since there is no way to maintain backwards compatibility now that that ship has sailed, I'm skeptical of it.
This is a pretty extreme statement, if it's not hyperbole I don't think we could ever agree. I'd much rather live in a world where bad decisions can be overturned/corrected in a sufficiently new version when a big enough problem is found. Backwards compatibility is important but I don't subscribe to the thought that any non-backwards compatible changes make a spec or a document worthless.
> I'm sure it will hit a good maxima at some point. But "is pretty easy to read and works pretty well for validation" was also fairly accurate for most XSD work. "People are already being very productive with it" was absolutely accurate. It was only when people started to stretch with it, that things got obnoxious. And we fell back on JSON not because it was somehow technically superior, but because it was technically easier.
Yeahhh I don't know if "pretty easy to read and works pretty well for validation" is true from XSD and the other XML ecosystem tools. Complexity was everywhere, and it introduced many bugs and critical security issues.
Also, I think that JSON is not only technically easier, it is technically simpler (in the Rich Hicky Simple vs Easy sense). JSON does not have potential cleverness built in, and that means it will likely never be what XML was (or at least all the complexity built on top will be very very opt-in).
Re: Arguing against using protobuffers
#144Earlier quoted context omitted.
The author isn’t trying to say either of those things, really. I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, th…
No, the author is showing off how smart they are, and in an unprofessional and nasty manner to boot. It's kind of you to gift the article with some genuinely thoughtful conclusions, but you're doing all the work there, not the original. As the grandparent says, it's easy to poke holes in something, especially when disregarding important requirements that influenced its design. The fact that the OP can't point to any…
The poster might be being kind, but their conclusions are spot on.
Re: Arguing against using protobuffers
#145> By far, the biggest problem with protobuffers is their terrible type-system. Fans of Java should feel right at home with protobuffers ...
Yeah right, because insulting only Protocol buffer and Google engineering won't be edgy enough for a blog post... :/
Re: Arguing against using protobuffers
#146Yet protobuf is probably the most compact, efficient and performant serialization method especially when saving bandwidth is important. I experimented with protofbuf, flatbuffers and messagepack and always found protobuf messages the most compact by a noticeable margin
That really depends on the nature of your data. The space saving in protobufs really comes from its variable-length zig-zag encoding of numeric fields (including lengths for arrays and strings). Tbh, i'm surprised msgpack didn't produce smaller outputs for you, because it really should (at the cost of being slower)
No way that msgpack can produce more compact messages than protobuf, it is schemaless and stores the element names inside the message. Here is a simple benchmark I have of an array of objects where each {id: int,x: int,y: int,rot: float}. An array of 20 elements with random values of such object would take in JSON 804 bytes, in mpack 538 bytes and in protobuf3 only 304 bytes, of course this changes if you put other values but as you see for this case protobuf is almost 60% of msgpack message size and it's around the same ration even if you make the array length 200 instead of 20.
Re: Arguing against using protobuffers
#147Put me firmly in the camp of "optional fields are bad." I believe all fields should be required. I come from an ONC/RPC background, which is the original UNIX RPC. Every iteration of an rpc would get versioned, and then you could write a conversion between versions, ex from V1 to V2, from V2 to V3, etc. This allowed for true backwards compatibility. The idea of "forwards compatibility" is a pipedream, in my opinion.…
If all fields are required, you cannot have middleware that processes multiple versions of the same protobuf, and every application has to be updated when a field is added, even if they do not use that field. This is one of the more important design goals underlying not only protobufs, but most of the non-language specific binary formatters.
> every application has to be updated when a field is added, even if they do not use that field
No, you maintain the older versions of the API. V1 of the API uses the V1 struct. V2 of the API uses the V2 struct, etc. Older applications maintain compatibility because it calls the older APIs, and you can convert between V1 to V2 and only keep one version of the API. Or, if you want, you can maintain both versions of the API, V1 and V2, at the expense of maintenance costs. But it's absolutely doable (we did it for decades).
Re: Arguing against using protobuffers
#148Hi there, I'm an actual author of Protocol Buffers :) I think Sandy's analysis would benefit from considering why Protocol Buffers behave the way they do rather than outright attacking the design because it doesn't appear to make sense from a PL-centric perspective. As with all software systems, there are a number of competing constraints that have been weighed that have led to compromises. - D P.S. I also don't beli…
I thought Kenton Varda was the author of protobufs?
Nope, I'm not the original author -- that would be Jeff Dean and Sanjay Ghemawat (also often credited with inventing things like MapReduce, BigTable, Spanner, ...). I wrote version 2 (a complete rewrite, but largely following the original design) and open sourced it. I stopped working on Protobuf about 8 years ago. Many others who have been on the Protobuf team since can certainly call themselves "authors".
Re: Arguing against using protobuffers
#149Versioning was the biggest disappointment for me. I just want a middleware layer that can handle clients of different versions reliably. Surely everyone has the same problem.
Avro has a solution to this that I always thought was quite elegant. Schemas are external (serialized data isn't self-describing) but the library exploits this to it easy to read and write old versions.
To quote the documentation:
The library provides "a resolving decoder, which accepts calls for according to a reader's schema but decodes data corresponding to a different (writer's) schema doing schema resolution according to resolution rules in the Avro specification."
Those resolution rules are defined here: https://avro.apache.org/docs/current/spec.html#Schema+Resolu...
The downside, of course, is performance.
Re: Arguing against using protobuffers
#150The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…
>experienced engineers don’t want to work with people who think like the author of this article - has opinions on interface design - isn't afraid to be wrong publicly - is brash on a personal blog I dunno, this is mostly positive. I'd have to see how well they'd adapt to the much different context and goals of one of our design reviews, but this isn't an immediate red flag. There's a million ways to be bad at a job.…
Brash on a personal blog is tricky. In this post's case, I'd worry that the blogger has a hard time separating technical deficiencies from professional incompetence. "Designed by amateurs" is an over-the-top and dubious claim.