Don't Use Protobuf for Telemetry

His point on protobuf-java library adding nontrivial bloat to his Java app is definitely valid. However, his other argument about protobuf wire format being inefficient is hard to square with decades of practical experience Google had with protobufs, which are used for literally everything, including telemetry, and high throughput, low latency applications. Sure, having to recursively precompute lengths before serialization is a bit of a hassle, but I wouldn’t call it expensive.

Re: Don't Use Protobuf for Telemetry

#5

Post by fwsgonzo » Wed, Dec 30, 2020, 7:36 PM UTC

We already have FlatBuffers, which allow you to send structs on the wire without reinventing anything. It sounds like it's a good match here. Protobuf is better if you care about the byte count on the wire and less about the CPU time to pack and unpack it.

Re: Don't Use Protobuf for Telemetry

#6

Post by as-j » Wed, Dec 30, 2020, 7:39 PM UTC

dariosalvi78 wrote: post #2

then what should we use? MQTT?

MQTT doesn’t specify the payload format. You can easily have a topic with the data in protobuf and another with json.

Re: Don't Use Protobuf for Telemetry

#7

Post by gregmac » Wed, Dec 30, 2020, 7:41 PM UTC

> Protobuf-java is a little heavy [...] Just depending on the library adds 1.6MB and nearly 700 classes before you even generate your own message classes.

By comparison, protobuf-net [1] is about 260KB and 68 classes. Python's [2] is a 1MB package download (with source).

Why's the Java one so big?

[1] https://github.com/protobuf-net/protobuf-net

[2] https://pypi.org/project/protobuf

Re: Don't Use Protobuf for Telemetry

#8

Post by janders » Wed, Dec 30, 2020, 7:41 PM UTC

I want to be a fan of Apache Avro (https://avro.apache.org/) so much for situations such as this. And while I like Avro as a standard, most of the implementations I have found (specifically in C/C++) are... lacking. I feel Avro would be a great fit for something like this close to zero overhead (assuming a pre-shared schema) but there is little to no support for pre-shared schema and the RPC part of the standard is not a great fit for telemetry. Maybe one day I'll make an Avro library I like, or contribute to an extant one.

Re: Don't Use Protobuf for Telemetry

#9

Post by sillysaurusx » Wed, Dec 30, 2020, 7:47 PM UTC

xyzzyz wrote: post #4

His point on protobuf-java library adding nontrivial bloat to his Java app is definitely valid. However, his other argument about protobuf wire format being inefficient is hard to square with decades of practical experience Google had with protobufs, which are used for literally everything, including telemetry, and high throughput, low latency applications. Sure, having to recursively precompute lengths before serial…

Signal boosting this. I am the last person to be swayed by “use our magical design that solves all problems,” but in the case of protobufs, it solves most of them.

I didn’t like it until I was forced to use it. Now I’m not sure I’ll ever go back.

There’s a handy code snippet to turn any protobuf message into JSON:

  from google.protobuf.json_format import MessageToJson
  import json
  from pprint import pprint as pp

  pp(json.loads(MessageToJson(msg)))

Examples of turning arbitrary tensorflow objects into JSON: https://twitter.com/theshawwn/status/1331625739993575427?s=2...

Re: Don't Use Protobuf for Telemetry

#10

Post by simonh » Wed, Dec 30, 2020, 7:51 PM UTC

fwsgonzo wrote: post #5

We already have FlatBuffers, which allow you to send structs on the wire without reinventing anything. It sounds like it's a good match here. Protobuf is better if you care about the byte count on the wire and less about the CPU time to pack and unpack it.

I'd not come across FlatBuffers before, but they do seem interesting for some use cases, especially when you're highly performance sensitive.

https://codeburst.io/json-vs-protocol-buffers-vs-flatbuffers...