Isn't this just a micro optimization? At what level does one need to care about this?
[deleted]
Don't Use Protobuf for Telemetry
11–20 of 195 posts
Re: Don't Use Protobuf for Telemetry
#12> Protobuf-java is a little heavy [...] Just depending on the library adds 1.6MB and nearly 700 classes before you even generate your own message classes. By comparison, protobuf-net [1] is about 260KB and 68 classes. Python's [2] is a 1MB package download (with source). Why's the Java one so big? [1] https://github.com/protobuf-net/protobuf-net [2] https://pypi.org/project/protobuf
Or just use the lite version?
It removes the heavy features like descriptors and reflection.
More details: https://developers.google.com/protocol-buffers/docs/referenc...
Re: Don't Use Protobuf for Telemetry
#13Don’t Use ~~Protobuf for~~ Telemetry
Re: Don't Use Protobuf for Telemetry
#14Google's own protobuf runtime is so bloated that even google themselves don't use it in many of their software, but rather nanopb[1], which is done by completely unrelated person to google.
Re: Don't Use Protobuf for Telemetry
#15Is this a thing people care about? From what I've seen of server-side Java applications (we have several at work) 1.6MB and 700 classes are lost in the noise of the endless list of Maven dependencies. I'm sure you can do it a lot more efficiently without using someone else's thing, but what exactly is being optimised for here?
Re: Don't Use Protobuf for Telemetry
#16His point on protobuf-java library adding nontrivial bloat to his Java app is definitely valid. However, his other argument about protobuf wire format being inefficient is hard to square with decades of practical experience Google had with protobufs, which are used for literally everything, including telemetry, and high throughput, low latency applications. Sure, having to recursively precompute lengths before serial…
Signal boosting this. I am the last person to be swayed by “use our magical design that solves all problems,” but in the case of protobufs, it solves most of them. I didn’t like it until I was forced to use it. Now I’m not sure I’ll ever go back. There’s a handy code snippet to turn any protobuf message into JSON: from google.protobuf.json_format import MessageToJson import json from pprint import pprint as pp pp(jso…
Why they use Java like code in Python? "MessageToJson" doesn't look idiomatic.
Re: Don't Use Protobuf for Telemetry
#17then what should we use? MQTT?
The post seems more like an argument against protobuf in general than against its use in telemetry. The main complaints seem to be that the library is big and that nested serialization is tricky because of design decisions.
In regards to binary size, using whatever serializing you're already doing for your main data seems like a win for telemetry. Including telemetry with your main data has pluses and minuses, but using the same serialization is generally fine, unless your serialization can fail, in which case it's hard to report on how many times telemetry serialization fails, because you might not be able to serialize it.
Re: Don't Use Protobuf for Telemetry
#18Isn't this just a micro optimization? At what level does one need to care about this?
It really depends on your scale and how many messages you are processing per second. For a lot of applications, you’re absolutely correct, but if you’re scale is sufficient, a “micro” optimization like this is actually a “macro” optimization. Also the author of this article works at DataDog and I suspect the number of messages they process each second falls under the sufficient category.
For example, suppose you are processing 1M messages per second and you can shave 1 byte off the message size, that shaves off 1MB/sec of data that needs to be processed. If you’re paying for network bandwidth or storing the messages, that saves you something like 2.6TB of data each month.
2.6TB/month is not likely to be a huge deal when it comes to cost savings, but if you keep scaling the messages/sec or the bytes/msg you can start to get some significant savings.
Now I used message size as an example, and the article focuses on processing time not message size, but the point still stands. When you can make a micro optimization for something that is done a very large number of times, there are not-insignificant gains to be had.
Re: Don't Use Protobuf for Telemetry
#19Are there any advantages of protobuf over using DER encoded ASN.1 ? The format looks very similar. It looks like they kind of reinvented the wheel here.
Re: Don't Use Protobuf for Telemetry
#20Isn't this just a micro optimization? At what level does one need to care about this?
I recommend HTTPS because it's easy to simulate and well understood. Also I like human-readable payloads.
Unless you need ridiculously low latencies and ridiculously high volume, you shouldn't worry at all.