Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't Use Protobuf for Telemetry

richardstartin.github.io

Post reply on HN
Print view View original

101–110 of 195 posts

Re: Don't Use Protobuf for Telemetry

#101

Post by ikiris »

rkagerer wrote: post #99
saberience wrote: post #97

Earlier quoted context omitted.

You don't work for Blizzard do you? When I worked there engineers (not my team thankfully) designed their own alternative to protobuf to try and save more bytes over the wire, which in my opinion was a really poor decision. Rather than get on with actually adding value, they ended up pushing back multiple other teams deadlines while adding almost no value. It was a classic "not built here" mentality and doing enginee…

If protobuf works for Google then it essentially works for 99.999% of every other company on the globe. I can't agree with this mindset. Another commenter here pointed out the Google implementation is 20x slower than others, and 1.6MB for this kind of task feels bloaty. Just because it meets Google's needs doesn't mean it's universally adequate.

Why exactly does a megabyte matter, unless you're trying to run on a 90's watch or embedded?

Re: Don't Use Protobuf for Telemetry

#102

Post by kentonv »

jasonzemos wrote: post #66
tijsvd wrote: post #54

Earlier quoted context omitted.

Things may have improved since, but the implementations are somehow very large and slow. Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages, and perhaps even for optional scalars. This may be optimal for larger documents, but not for smallish messages (my use case was market data and trading instructions). I measured certain small messages, where an e…

Since all other comments appear to contradict you and make apologies from authority (after all, Google can't do anything wrong, right?) I'd like to just reassure you with my 20 years of experience developing C (and the last 10 with C++) in network and system software: that any library -- any library -- that forces internal dynamic memory upon its user smells bad. It's not a universal condemnation, but it begs the que…

Unless you are targeting a constrained embedded system, this kind of anti-memory-allocation thinking is counter-productive. Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers.

In C, dealing with memory allocation is such a pain that C programmers still tend to avoid it. But C++, especially post-C++11, makes dynamic memory allocation much, much easier to deal with. (And GC'd languages, obviously, are easier still.) There is still a performance cost, obviously, but that cost almost never matters in application programming use cases, and is even negligible in many systems programming cases.

I do think Protobuf does too much allocation, but saying that libraries should not allocate memory at all is an outdated view.

Re: Don't Use Protobuf for Telemetry

#103

Post by FpUser »

ikiris wrote: post #101
rkagerer wrote: post #99

Earlier quoted context omitted.

If protobuf works for Google then it essentially works for 99.999% of every other company on the globe. I can't agree with this mindset. Another commenter here pointed out the Google implementation is 20x slower than others, and 1.6MB for this kind of task feels bloaty. Just because it meets Google's needs doesn't mean it's universally adequate.

Why exactly does a megabyte matter, unless you're trying to run on a 90's watch or embedded?

Maybe because "20X slower"

Re: Don't Use Protobuf for Telemetry

#104

Post by sebastos »

kentonv wrote: post #98
sebastos wrote: post #90

Earlier quoted context omitted.

Damned if you're not the most patient self-advocate for an open source project I've ever seen. I feel like I can always count on finding The Kenton Varda in the comments whenever a protobuf discussion comes up, graciously discussing design trade-offs and dispelling misconceptions. Did you ever consider looking for official / commercial backing for Cap N Proto? It seems demonstrably better than all the alternatives, b…

Hah, thanks. > Did you ever consider looking for official / commercial backing for Cap N Proto? Well, I don't think Cap'n Proto is marketable (for any kind of profit) on its own. It's one of those things that has to be free and open source to get any adoption. So a corporate backer would need to get some indirect benefit from its development and adoption. Also, at this point my goal in building and maintaining Cap'n…

Thanks for the insight, that makes sense!

Perhaps this is too unhip, but I wonder - have you ever dealt with the OMG's Data Distribution Service (DDS)? I used to think it was too fuddy-duddy to even glance at, but the recent adoption of DDS by ROS2 made me take another look. DDS seems to have a lot of interesting properties, particularly the QoS and discovery mechanisms. What made me think of it is that DDS is really a free and open standard, but there are a number of commercial entities making money off of it. They provide DDS implementations for free, and then they make money by offering professional consulting and support. Maybe a Cloudflare-backed capnp could end up looking the same way. Anyway, I'm sure you have your head around the possibilities and tradeoffs, not sure why I'm rambling.

Re: Don't Use Protobuf for Telemetry

#105

Post by dweis »

FpUser wrote: post #103
ikiris wrote: post #101

Earlier quoted context omitted.

Why exactly does a megabyte matter, unless you're trying to run on a 90's watch or embedded?

Maybe because "20X slower"

These are simply tradeoffs. For example, one can make very fast protobuf parsers/serializers if you prioritize that over the ergonomics of the generated code.

Re: Don't Use Protobuf for Telemetry

#106

Post by kentonv »

WorldMaker wrote: post #96
kentonv wrote: post #76

Earlier quoted context omitted.

> It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully t…

> Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. They just did something that worked well enough, and it worked well enough that it was rapidly adopted by th…

Yes, I agree with that. If Protobuf hadn't been a reasonably-good design for Google-scale distributed systems, it wouldn't have won (within the company, or beyond). So regardless of how that design came about, we can say it is a reasonably good design overall. But only in aggregate -- not every individual decision can be assumed to be great on its own.

Re: Don't Use Protobuf for Telemetry

#107

Post by jasonzemos »

haberman wrote: post #80
jasonzemos wrote: post #66

Earlier quoted context omitted.

Since all other comments appear to contradict you and make apologies from authority (after all, Google can't do anything wrong, right?) I'd like to just reassure you with my 20 years of experience developing C (and the last 10 with C++) in network and system software: that any library -- any library -- that forces internal dynamic memory upon its user smells bad. It's not a universal condemnation, but it begs the que…

> make apologies from authority (after all, Google can't do anything wrong, right?) That's not my position at all. In my other comment ( https://news.ycombinator.com/item?id=25586447 ) I explain how I've spent 10 years trying to improve on protobuf C++ precisely because I agree that some of these limitations are unnecessary. > No zero-copy for networking? Forced internal heap allocations with only this arena feature…

> I suppose it depends what you are comparing it to. Almost every JSON library has the same limitations you mentioned, and yet many people find JSON useful for network applications.

JSON is a human-readable format which is hugely advantageous to develop and operate in many settings. Protobufs doesn't have that advantage. Yet we're paying all of the same costs to structure the data with both. That's enough to posit JSON as a net winner over protobufs.

> Unfortunately protobuf C++ can't live fully within this arena model while it uses std::string for accessors.

Forcing std::string as a container for core components of a networking API (specifically for an accessor) is an exemplary demonstration of a lack of seriousness in a library.

Re: Don't Use Protobuf for Telemetry

#108

Post by vvanders »

kentonv wrote: post #102
jasonzemos wrote: post #66

Earlier quoted context omitted.

Since all other comments appear to contradict you and make apologies from authority (after all, Google can't do anything wrong, right?) I'd like to just reassure you with my 20 years of experience developing C (and the last 10 with C++) in network and system software: that any library -- any library -- that forces internal dynamic memory upon its user smells bad. It's not a universal condemnation, but it begs the que…

Unless you are targeting a constrained embedded system, this kind of anti-memory-allocation thinking is counter-productive. Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers. In C, dealing with memory allocation is such a pain that C programmers still tend to avoid it. But C++, especially post…

Constrained embedded systems cover a broad range of things up to and including your phone. There's few things I hate to see more than a flat profile from memory allocation or cache misses.

The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation sparse.

[1] https://google.github.io/flatbuffers/

Re: Don't Use Protobuf for Telemetry

#109

Post by bastawhiz »

Ignoring the bits about Java specifically, one thing that jumps out at me (while reading through the comments) is the distinction between use cases. It seems that the wire format puts some extra burden on the encoding side, but has the benefit of making decoding nice.

In a situation where you're using protobufs internally, you're both encoding and decoding the messages. CPU hours aren't fungible, but you're paying a small latency cost somewhere in your system between when the data is produced and when it can be consumed.

In the case of telemetry, such as with Datadog, you pretty much only encode the data. That is, if I'm a Datadog customer, I'm literally never decoding the encoded data on my own servers. That being the case, it would seem that the argument is that if encoding performance is a desirable property, protobufs add overhead that doesn't offer any meaningful tradeoff to the user ("you're using more CPU on my server and I don't see tangible benefits").

Is this an incorrect interpretation?

Re: Don't Use Protobuf for Telemetry

#110

Post by nijave »

cpascal wrote: post #18
ketamine__ wrote: post #3

Isn't this just a micro optimization? At what level does one need to care about this?

It really depends on your scale and how many messages you are processing per second. For a lot of applications, you’re absolutely correct, but if you’re scale is sufficient, a “micro” optimization like this is actually a “macro” optimization. Also the author of this article works at DataDog and I suspect the number of messages they process each second falls under the sufficient category. For example, suppose you are…

I think this was for client/application instrumentation where they want to try to have little-to-no overhead for including the datadog agent that ships telemetry back to Datadog's service
Post reply on HN