Protobuffers always seemed like an interesting approach but every time I've tried to use in a prototype I've ended up deciding against the added overhead of including them. And then I use JSON because I simply wanted versioned serialisation when messaging or persisting data.
Arguing against using protobuffers
131–140 of 307 posts
Re: Arguing against using protobuffers
#132Yet protobuf is probably the most compact, efficient and performant serialization method especially when saving bandwidth is important. I experimented with protofbuf, flatbuffers and messagepack and always found protobuf messages the most compact by a noticeable margin
That's the core of the author's argument. Protobuffers optimize for something besides usability and maintainability, because Google cares more about incremental performance than developer-friendliness. Which is a fine thing to care about at Google's scale, but maybe others' calculations should be different.
That doesn't seem like the author's main argument - they say: "Protobuffers were obviously built by amateurs because they offer bad solutions to widely-known and already-solved problems."
Re: Arguing against using protobuffers
#133I spent 2.5 years at Google, and most of what I did was pushing one protobuf from one place to another :) - and I loved it... Honestly though, you can complain all day, but some of the decisions made in the list you presented most likely come from experience (daily) not as a user of protobufs (which I simply was), but someone that had to support a plethora of compression formats, how protobufs gets stored in the diff…
I only interned at Google but I also really liked protocol buffers, somehow it seems very structured and defined to me, and better than Java. But this could just be that I haven't worked with better things... Lol
Re: Arguing against using protobuffers
#134Yet protobuf is probably the most compact, efficient and performant serialization method especially when saving bandwidth is important. I experimented with protofbuf, flatbuffers and messagepack and always found protobuf messages the most compact by a noticeable margin
That really depends on the nature of your data. The space saving in protobufs really comes from its variable-length zig-zag encoding of numeric fields (including lengths for arrays and strings).
Tbh, i'm surprised msgpack didn't produce smaller outputs for you, because it really should (at the cost of being slower)
Re: Arguing against using protobuffers
#135Earlier quoted context omitted.
Nope. There are similar formats designed to be rapidly serializable and back that outperform protobufs on this front. * cap'n proto https://capnproto.org/ * flatbuffers https://google.github.io/flatbuffers/ * hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/ You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beat…
Am I missing something, or do cap'n proto and flatbuffers not support mapping types? Protobuf has disadvantages, but mapping types are something I frequently make use of. losing out on them is a big deal
Protobuf did not support maps for a long time. In practice using a list of key/value pairs works fine for most use cases. Sending the message over the wire is obviously always O(n); if you need fast inserts/lookups in code, then it usually makes sense to convert to and from the appropriate in-memory data structure for that purpose.
That said, I think maps are a fine feature for a serialization format to have. But, I haven't gotten around to adding them in Cap'n Proto because I have yet to hit a use case where I wasn't happy with a list of key/value pairs.
Re: Arguing against using protobuffers
#136The main point that people are missing is that experienced engineers don’t want to work with people who think like the author of this article. Protocol Buffers are not wrong, they simply have constraints, advantages, and disadvantages. No language, binary format, text format, etc is free from advantages and disadvantages. All of them have different use cases. If you are building a system where your data can be descri…
The author isn’t trying to say either of those things, really. I think their real point was something like: many companies that have an Enterprise Service Bus architecture for their pile of polyglot microservices, have a dogma for the encoding of the data flowing over the Bus. And Protobufs, though good for some use-cases, are a particularly bad dogma to be stuck with. That is, when they don’t work for a use-case, th…
No, the author is showing off how smart they are, and in an unprofessional and nasty manner to boot.
It's kind of you to gift the article with some genuinely thoughtful conclusions, but you're doing all the work there, not the original.
As the grandparent says, it's easy to poke holes in something, especially when disregarding important requirements that influenced its design. The fact that the OP can't point to any implementation of the "right" way to do things is telling.
Re: Arguing against using protobuffers
#137Earlier quoted context omitted.
Nope. There are similar formats designed to be rapidly serializable and back that outperform protobufs on this front. * cap'n proto https://capnproto.org/ * flatbuffers https://google.github.io/flatbuffers/ * hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/ You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beat…
Am I missing something, or do cap'n proto and flatbuffers not support mapping types? Protobuf has disadvantages, but mapping types are something I frequently make use of. losing out on them is a big deal
[deleted]
Re: Arguing against using protobuffers
#138If protobuffers are wrong, then I don’t want to be right.
Re: Arguing against using protobuffers
#139I will say one thing. gzipped JSON is damn efficient over the wire. It elegantly solves the problem that field names are repeated in every object. The only problem is that adding compression and a serializationd/deserialization step cost CPU time. However if you have this problem then you're often better off by using something like cap'n proto or flatbuffers which do not have an extra serdes step. If you consider thi…
The problem with JSON as a wire/interop format is the lack of any sort of schema. Formats like proto are nice because if you can unmarshall the bytes successfully, you can be reasonably sure you've got a valid message and reason about its contents. A deserialized JSON object can literally be or contain anything.
Proto doesn't really have a schema. You can encode a proto of type X and tell your program to decode it as type Y, and it might work. Might not, but might. You can easily imagine messages that are isomorphic on the wire. E.g.
message X { optional string foo = 1; } message Y { optional X foo = 1; }
Encoded buffers of X will decode as Y just fine.
Re: Arguing against using protobuffers
#140Earlier quoted context omitted.
Nope. There are similar formats designed to be rapidly serializable and back that outperform protobufs on this front. * cap'n proto https://capnproto.org/ * flatbuffers https://google.github.io/flatbuffers/ * hdf5 (for machine learning/numerical analysis/finance) https://support.hdfgroup.org/HDF5/ You're just not going to beat these formats in serialization speed with anything (just mmap the file, use. Good luck beat…
Am I missing something, or do cap'n proto and flatbuffers not support mapping types? Protobuf has disadvantages, but mapping types are something I frequently make use of. losing out on them is a big deal
No, you're right. The canonical way to store a map in capnp is to use a List>. Personally it doesn't bother me that much (mostly from an aesthetic standpoint) because it is not meant as an in-memory, working data structure.
What does bother me is that in the above generic type, "key" and "value" must be pointer types (i.e. non-scalar). The reasons are not unreasonable but it's definitely obnoxious in practice.