Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't Use Protobuf for Telemetry

richardstartin.github.io

Post reply on HN
Print view View original

141–150 of 195 posts

Re: Don't Use Protobuf for Telemetry

#141

Post by kentonv »

jasonzemos wrote: post #138
kentonv wrote: post #135

Earlier quoted context omitted.

You can debate the design decisions all you want (and I agree `std::string` was a poor choice here)... but it's kind of absurd to say that a technology underpinning products used by billions of people every day is not "serious".

Why is that absurd? Products used by billions of people may just as well not have serious requirements. I beg not to debate semantics, indeed my own fault, but in the world of network software requirements can start to get very serious, straight through to userspace protocol stacks on DMA'ed device buffers. Would you go through all of the effort to rig an application with userspace networking only to return its resul…

If you are arguing that there exist use cases for which Protobuf is ill-suited, then fine, I obviously agree.

But you seem to be arguing that Protobuf is bad because it is not well suited to certain use cases, dismissing all other use cases as "non-serious". That is offensive.

Re: Don't Use Protobuf for Telemetry

#142

Post by girvo »

kentonv wrote: post #62
tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Right, they were designed for use in Google's servers, where binary size is mostly irrelevant, while speed and features (e.g. reflection) are valued. "Lite mode" (not mentioned in the article, for some reason) optimizes for code size instead. Admittedly, it's…

HN is one of the few places you can talk smack about a companies tooling and have the person who wrote it reply!

Lite mode is neat, I hadn’t seen that. That’s super useful for my use case, cheers. Any gotchas with it to be aware of?

Re: Don't Use Protobuf for Telemetry

#143

Post by sargun »

I work with an application that spends something like 70-80% of its time in serialization / deserialization code. This is mostly due to heap allocation / deallocation (in Golang) of objects, and the RPC stacks in use not doing allocation arenas, reuse, or similar.

The "state of the art" in de(serialization) isn't awesome. :(.

Re: Don't Use Protobuf for Telemetry

#144

Post by a_t48 »

jasonzemos wrote: post #132
kentonv wrote: post #102

Earlier quoted context omitted.

Unless you are targeting a constrained embedded system, this kind of anti-memory-allocation thinking is counter-productive. Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers. In C, dealing with memory allocation is such a pain that C programmers still tend to avoid it. But C++, especially post…

> Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers. Allow me to unpack this, because the object interface that all of these serialization schemes present is what increases complexity. It makes sense that the simple implementation would tend toward dynamic memory and not further compound such…

> These libraries are far too pro-active in deserializing incoming data into native object structures.

Sometimes a native object structure is what you want.

Re: Don't Use Protobuf for Telemetry

#145

Post by vvanders »

kentonv wrote: post #116
vvanders wrote: post #108

Earlier quoted context omitted.

Constrained embedded systems cover a broad range of things up to and including your phone. There's few things I hate to see more than a flat profile from memory allocation or cache misses. The good news is that flatbuffers[1] is a reasonable replacement for most of my use cases. In particular being able to mmap() them directly is a wonderful thing that you can't do with protobufs in addition to being very allocation…

> Constrained embedded systems cover a broad range of things up to and including your phone. No, modern phones are certainly not constrained in the way I meant, and I don't think you could call them "embedded" either. The common programming languages used on phones are very memory-allocation-friendly. > There's few things I hate to see more than a flat profile from memory allocation or cache misses. I think you may b…

> The common programming languages used on phones are very memory-allocation-friendly.

I would hardly classify Dalvik or ART as "allocation friendly", they don't perform escape analysis and if you do it constantly you'll be in a world of constant hard GC pauses. Multiple times over the years I've had to build free-lists in Java to avoid this specific problem.

Same for C++ if you use one of the built-in generic memory allocators. The fastest new/delete are the ones that you don't call.

For what it's worth I tend to agree with the grand-parent thread. The lack of awareness of allocations, cache-invalidation via indirection are a significant contributor to why we see software clawing back hardware wins across the years on these platforms.

Re: Don't Use Protobuf for Telemetry

#146

Post by a_t48 »

kentonv wrote: post #64
tijsvd wrote: post #54

Earlier quoted context omitted.

Things may have improved since, but the implementations are somehow very large and slow. Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages, and perhaps even for optional scalars. This may be optimal for larger documents, but not for smallish messages (my use case was market data and trading instructions). I measured certain small messages, where an e…

> AFAIK the C++ implementation would always allocate on the heap for nested messages FWIW if you reuse the same message object for multiple parsings, it will re-use the sub-objects as well, thus amortizing away the allocation cost. Parsing the same message into the same object twice should do zero allocations on the second parse. This is the intended way to use Protobuf for small-size messages. Apparently the C++ imp…

I wish I had known this a few years ago, I could have saved some allocations. :)

Re: Don't Use Protobuf for Telemetry

#147

Post by RhodesianHunter »

naniwaduni wrote: post #122
ikiris wrote: post #101

Earlier quoted context omitted.

Why exactly does a megabyte matter, unless you're trying to run on a 90's watch or embedded?

Well, transmitting a megabyte also takes a full second at 8 Mbps.

Nobody's transmitting this, it's the size of the library doing the proto serialization/deser

Re: Don't Use Protobuf for Telemetry

#148

Post by RhodesianHunter »

ashtonkem wrote: post #29
jeffbee wrote: post #21

Earlier quoted context omitted.

That was kinda my response as well. If telemetry altogether is using more than 1% of your cpu time you have an architectural problem perhaps.

My experience points to the cost of storing metrics far outweighing the cost of emitting metrics in all but the most extreme of circumstances.

Yes, but in this case the author is an employee of DataDog, presumably working on their agents or clients and so is probably acutely focused on not impacting the performance of their customer's systems.

Re: Don't Use Protobuf for Telemetry

#149

Post by jasonzemos »

kentonv wrote: post #141
jasonzemos wrote: post #138

Earlier quoted context omitted.

Why is that absurd? Products used by billions of people may just as well not have serious requirements. I beg not to debate semantics, indeed my own fault, but in the world of network software requirements can start to get very serious, straight through to userspace protocol stacks on DMA'ed device buffers. Would you go through all of the effort to rig an application with userspace networking only to return its resul…

If you are arguing that there exist use cases for which Protobuf is ill-suited, then fine, I obviously agree. But you seem to be arguing that Protobuf is bad because it is not well suited to certain use cases, dismissing all other use cases as "non-serious". That is offensive.

I asked you not to argue semantics, but you're doing it anyway. I didn't respond here to argue protobufs actually is bad, since all software is suited to something for somebody somewhere.

In fact, I can just do that now: Protobufs is bad. You seem to agree, because you haven't even advocated for it -- just flatbuffers and Cap'n Proto and the phonebook of alternatives. Listen, I don't care who you've worked for or what you've done. As a scientist and engineer I'm totally ready and able to entertain critiques of my prior works without being offended. I'm offended that this is the quality of your participation here.

Re: Don't Use Protobuf for Telemetry

#150

Post by alexhutcheson »

jmkjaer wrote: post #55
jeffbee wrote: post #22

Earlier quoted context omitted.

I don’t think that’s true in google3. People often mistake the scale of Google’s internal code with respect to their visible external code.

For anyone unfamiliar, Google3 is apparently Google's monorepo, according to this HN comment: https://news.ycombinator.com/item?id=17123620

https://research.google/pubs/pub45424/
Post reply on HN