Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't Use Protobuf for Telemetry

richardstartin.github.io

Post reply on HN
Print view View original

171–180 of 195 posts

Re: Don't Use Protobuf for Telemetry

#171

Post by tijsvd »

saberience wrote: post #97
tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

You don't work for Blizzard do you? When I worked there engineers (not my team thankfully) designed their own alternative to protobuf to try and save more bytes over the wire, which in my opinion was a really poor decision. Rather than get on with actually adding value, they ended up pushing back multiple other teams deadlines while adding almost no value. It was a classic "not built here" mentality and doing enginee…

No I don't work for Blizzard. When I did this, we were using a very fast custom wire format, but entirely hand-coded. Say flatbuffers without the code generation part. And having massive trouble with schema evolution.

Another group had already evaluated pb and found it way too slow. They had designed something similar but faster. I wrote my implementation of pb to prevent this, and showed pb could be fast enough. It was definitely the right choice at the time, as it gave us C++ speed close to the old format, plus easy interop with other languages.

> If protobuf works for Google then it essentially works for 99.999% of every other company on the globe.

Uh.. no. Google is a massive company, but if you browse the comments in this thread, you'll find multiple remarks like "this was built for Google's servers". Google have specific use cases, and they build software for that. The software may well be lacking for other use cases. I can totally imagine Blizzard wanting to write their own implementation, think of the benefits of reducing parse time in a multiplayer server.

Re: Don't Use Protobuf for Telemetry

#172

Post by ganafagol »

Isn't there an easy solution for this? A varint representation does not mandate that you use its variable length. Just pad with leading zeros and you have a fixed length.

If I understand the varint format right, its groups of 7 bits, stuffed into one byte each. Alright, so let's just always use 4 bytes, which gives you a value range from 0 to 2^28-1 for the length, that's up to 256M which should suffice if you are that concerned about telemetry speed. Now your length field is always 4 bytes. Even if you could shrink it to 1 or 2 most of the time, but since you care more about encoding speed than size on the wire that should be fine. Nothing forces you to encode value 42 in 1 byte. Just use 4.

Am I missing something? Honest question.

Re: Don't Use Protobuf for Telemetry

#173

Post by haberman »

kentonv wrote: post #88
kentonv wrote: post #76

Earlier quoted context omitted.

> It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully t…

> The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. And just to be clear, I don't think this is bad. On the contrary, I think Protobuf won because it did a wide variety of things "pretty well" while moving quickly and solving real problems. This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out…

> This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out something that works and running with it to solve real problems.

I think this is a really important point. I don't think I've ever told you this before, but I am really impressed by how quickly you turned out proto2. While there are some things here and there that we wish we could change, a lot of it holds up really well. A lot of decisions I made in upb early on, where I thought I was improving on proto2, actually turned out to be bad ideas and the proto2 design was better.

Re: Don't Use Protobuf for Telemetry

#174

Post by [deleted] »

kentonv wrote: post #64
tijsvd wrote: post #54

Earlier quoted context omitted.

Things may have improved since, but the implementations are somehow very large and slow. Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages, and perhaps even for optional scalars. This may be optimal for larger documents, but not for smallish messages (my use case was market data and trading instructions). I measured certain small messages, where an e…

> AFAIK the C++ implementation would always allocate on the heap for nested messages FWIW if you reuse the same message object for multiple parsings, it will re-use the sub-objects as well, thus amortizing away the allocation cost. Parsing the same message into the same object twice should do zero allocations on the second parse. This is the intended way to use Protobuf for small-size messages. Apparently the C++ imp…

[deleted]

Re: Don't Use Protobuf for Telemetry

#175

Post by sagarm »

Animats wrote: post #157
sebastos wrote: post #100

Earlier quoted context omitted.

Well, I started to read the article, thinking that it might refer to sending, you know, telemetry. The hackernews link was titled like this: >Don't Use Protobuf for Telemetry so I thought I had a shot of reading something relevant to me. So I opened up the article: >Protobuf needs no introduction, but this post argues that you shouldn’t use it for telemetry. The basic premise of this post is that a good telemetry lib…

Yes. Just recently I discovered that some web types refer to generating HTML from some other representation as "rendering". To graphics people, and artists, rendering is taking a description of something and turning it into a directly viewable picture. HTML is a long, long way from displayed pixels. Come on. There's a big industry out there working to push game and entertainment content through the GPU and onto the s…

Rendering is what happens when fat is cooked slowly over low heat and becomes a liquid, rather than crisping up. What pixels have to do with that, I have no idea.

Re: Don't Use Protobuf for Telemetry

#176

Post by shaicoleman »

brentru wrote: post #35

The author mentioned msgpack, what else would someone suggest for low-latency use cases?

Serialization/deserialization is rarely the bottleneck, so don't prematurely optimise for it.

The candidates for fast serialization/deserialization that have implementations for all major programming languages are:

* Plain JSON using the simdjson library, optionally in NDJSON format (line separated), optionally compressed with LZ4.

* CBOR (like msgpack but with more datatypes/features)

* msgpack

* Protocol Buffers (optionally with gRPC)

Re: Don't Use Protobuf for Telemetry

#177

Post by anuraaga »

I work on OpenTelemetry Java, including a Java Agent that is originally a fork from Datadog's excellent base which they contributed to the project.

Protobuf definitely has a sizable footprint, but I guess it's still ok compared to other popular libraries like Guava and Jackson. I suspect Richard is saving an article for gRPC+Netty which is extremely large :-) Lots of classes, and every shaded Netty on a classpath has its own arena of pooled buffers.

Wire format I like well enough though. I think protobuf does provide good expressiveness to allow nesting if that fits the bill, or more flat structures, where it is a joy to output repeated fields completely independently and even interleaved in a stream.

It's unclear to me if protobuf itself is a bad wire format for telemetry or heavy use of nesting is. The OpenTelemetry Protocol is highly nested to be completely denormalized. This reduces data on the wire (or not so much compared to gzip, but can be considered a hand-coded gzip), but it means all data has to be traversed before any can be sent to compute the length prefix. I don't know which wins in practice - I've just never seen any comparison of the exact same data with different formats. I'm hoping the CPU cache alleviates the double traversal though. But a less nested format in protobuf would easily be possible.

Re: Don't Use Protobuf for Telemetry

#179

Post by marco_craveiro »

tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

Its a shame you can't share the code, we could really do with a modern C++ implementation. Having said that, protozero looks very interesting [1], need to find some time to look into it.

[1] https://github.com/mapbox/protozero

Re: Don't Use Protobuf for Telemetry

#180

Post by tailspin2019 »

girvo wrote: post #142
kentonv wrote: post #62

Earlier quoted context omitted.

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Right, they were designed for use in Google's servers, where binary size is mostly irrelevant, while speed and features (e.g. reflection) are valued. "Lite mode" (not mentioned in the article, for some reason) optimizes for code size instead. Admittedly, it's…

HN is one of the few places you can talk smack about a companies tooling and have the person who wrote it reply! Lite mode is neat, I hadn’t seen that. That’s super useful for my use case, cheers. Any gotchas with it to be aware of?

> HN is one of the few places you can talk smack about a companies tooling and have the person who wrote it reply!

Not just that, I suspect it’s a sign of HN’s unique culture that the “smack talk” seems not to be taken personally and instead a highly nuanced and interesting discussion ensues!

Post reply on HN