Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't Use Protobuf for Telemetry

richardstartin.github.io

Post reply on HN
Print view View original

31–40 of 195 posts

Re: Don't Use Protobuf for Telemetry

#31

Post by tijsvd »

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively.

Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at least some implementation available.

Then, the author laments the wire format itself for having varint-encoded length prefixes, which can not be fixed up. That is true, but it's not that much of a problem. Most straightforward is to simply go through nested data multiple times, once to calculate the length, and again for the actual encoding (and then again and again for deeper nesting).

What makes this bearable is the fact that data will be mostly loaded into L2 cache (L1 for smaller messages) on the first pass, which makes the next pass much faster.

The story breaks down for large, deeply nested messages, but then, the topic here is telemetry which I would expect to consist of a stream of small, shallow messages.

Re: Don't Use Protobuf for Telemetry

#32

Post by tyingq »

fwsgonzo wrote: post #5

We already have FlatBuffers, which allow you to send structs on the wire without reinventing anything. It sounds like it's a good match here. Protobuf is better if you care about the byte count on the wire and less about the CPU time to pack and unpack it.

It also appears to be 236kb and 23 classes for the java api, versus 1.6MB and 700 classes for protobufs.

Re: Don't Use Protobuf for Telemetry

#34

Post by thisiscorrect »

tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated?

[edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?

Re: Don't Use Protobuf for Telemetry

#36

Post by ipsum2 »

intricatedetail wrote: post #16
sillysaurusx wrote: post #9

Earlier quoted context omitted.

Signal boosting this. I am the last person to be swayed by “use our magical design that solves all problems,” but in the case of protobufs, it solves most of them. I didn’t like it until I was forced to use it. Now I’m not sure I’ll ever go back. There’s a handy code snippet to turn any protobuf message into JSON: from google.protobuf.json_format import MessageToJson import json from pprint import pprint as pp pp(jso…

Why they use Java like code in Python? "MessageToJson" doesn't look idiomatic.

The official protobuf library for Python is not pythonic at all.

Re: Don't Use Protobuf for Telemetry

#37

Post by twic »

simonh wrote: post #10
fwsgonzo wrote: post #5

We already have FlatBuffers, which allow you to send structs on the wire without reinventing anything. It sounds like it's a good match here. Protobuf is better if you care about the byte count on the wire and less about the CPU time to pack and unpack it.

I'd not come across FlatBuffers before, but they do seem interesting for some use cases, especially when you're highly performance sensitive. https://codeburst.io/json-vs-protocol-buffers-vs-flatbuffers...

AFAIK, the three main options in this subspace are FlatBuffers, Cap'n Proto, and SBE.

Here's a comparison from a Rust perspective: https://speice.io/2019/09/binary-format-shootout.html

And from the Cap'n Proto author: https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-...

Re: Don't Use Protobuf for Telemetry

#38

Post by pb123456 »

In many cases you do indeed want or need to use Protobuf for telemetry or other time-series data. The Protobuf RecordIO / TFRecord formats are very unhelpful for this use case.

Enter Protobag: https://github.com/StandardCyborg/protobag

Protobag is a library that helps you write and read time-series and other collections of Protobuf data using the standard Tar and Zip archives as containers. Protobag also includes code for leveraging self-describing messages to embed schemas in archives so that you never have data on disk that you can't read. Hope somebody finds Protobag helpful.

Re: Don't Use Protobuf for Telemetry

#39

Post by Nvorzula »

Aaargh20318 wrote: post #19

Are there any advantages of protobuf over using DER encoded ASN.1 ? The format looks very similar. It looks like they kind of reinvented the wheel here.

To my understanding they're both [type, length, content] encoded formats, so they're similar there at least.

In my experience people really quite enjoy the IDL aspect of protobuf as the killer feature, and I am not sure any such compiler exists for ASN1.

Re: Don't Use Protobuf for Telemetry

#40

Post by gabereiser »

thisiscorrect wrote: post #34
tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated? [edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?

Probably a complaint of the codegen and it’s wacky name mangling.
Post reply on HN