Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't Use Protobuf for Telemetry

richardstartin.github.io

Post reply on HN
Print view View original

81–90 of 195 posts

Re: Don't Use Protobuf for Telemetry

#81

Post by seppel »

haberman wrote: post #61
tijsvd wrote: post #54

Earlier quoted context omitted.

Things may have improved since, but the implementations are somehow very large and slow. Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages, and perhaps even for optional scalars. This may be optimal for larger documents, but not for smallish messages (my use case was market data and trading instructions). I measured certain small messages, where an e…

> Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages This is no longer the case if you use arenas: https://developers.google.com/protocol-buffers/docs/referenc... > and perhaps even for optional scalars This has never been the case, except for string fields where std::string forces us to allocate. Ideally we will eventually use std::string_view for st…

> This has never been the case, except for string fields where std::string forces us to allocate.

I'm was quite surprised you didnt offer your own stringview implementation (or something similar) the last time I looked at protobuf. I'd naively assume that inside Google this could be quite a low-efford high-reward optimization.

Re: Don't Use Protobuf for Telemetry

#82

Post by jasonzemos »

tijsvd wrote: post #78
jasonzemos wrote: post #66

Earlier quoted context omitted.

Since all other comments appear to contradict you and make apologies from authority (after all, Google can't do anything wrong, right?) I'd like to just reassure you with my 20 years of experience developing C (and the last 10 with C++) in network and system software: that any library -- any library -- that forces internal dynamic memory upon its user smells bad. It's not a universal condemnation, but it begs the que…

> No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications. That's a bit harsh. Protobufs deliver smaller wire size than any of the newer "zero-copy" formats. And many receivers of zero-copy formats will... copy the data into some internal representation. If your protobuf implementation delivers classes…

CBOR also delivers optimal wire sizes with its variable length encoding. Ceteris paribus, there's no technical advantage to using protobufs. The choice is almost always non-technical (business, platform, partnership, etc) or simple naivete. It's not harsh, it's just engineering.

Re: Don't Use Protobuf for Telemetry

#83

Post by haberman »

kentonv wrote: post #76
WorldMaker wrote: post #25

Earlier quoted context omitted.

It seems easy to square the practical experience of protobufs at Google versus the tradeoffs encountered in the wire format when you take into account how many Google engineers have directly contributed to the diaspora of related formats such as CapnProto, flatbuffers, msgpack, etc with different tradeoffs (especially for smaller scales than Google). It seems clear enough that protobufs were optimized in a scale that…

> It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully t…

Hey that reminds me, I've always wondered: how did it come to be that delimited encoding "won" for sub-messages and group encoding was deprecated? A lot of these problems would go away if things had gone the other way.

Groups can be encoded in one pass, they are efficient to decode (unless you were trying to skip the sub-message, a la LazyField), and they don't have the string/message ambiguity in UnknownFieldSet that messages have.

Do you remember how that came to pass? Maybe some of this happened during the evolution of proto1, which was before my time.

Re: Don't Use Protobuf for Telemetry

#84

Post by sebastos »

Animats wrote: post #46
haberman wrote: post #44

Earlier quoted context omitted.

The author of this piece is wanting to stream possibly multi-MB messages with sub-linear memory use. See more in our Twitter conversation here: https://twitter.com/richardstartin/status/134406813297822105...

"possibly multi-MB messages" Megabytes? By "telemetry" does he mean "getting info on what our pumps out in the oil field are doing" or "snooping on users of a program"?

This over-assuming-of-shared-context is one of my pet peeves on this site. Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow.

Of course, I'm also annoyed that some computer science PhD apparently googled around for a cooler name for multi-dimensional arrays and ruined the term "Tensor" forevermore.

Re: Don't Use Protobuf for Telemetry

#85

Post by Cloudef »

brandmeyer wrote: post #24
Cloudef wrote: post #14

Google's own protobuf runtime is so bloated that even google themselves don't use it in many of their software, but rather nanopb[1], which is done by completely unrelated person to google. 1: https://jpa.kapsi.fi/nanopb/

> even google themselves don't use it in many of their software Citation needed. Granted its a big company, so its certainly possible that some teams I'm unaware of were using it. But I never ran into one. Besides, nanopb doesn't answer the critique of the article. Nanopb cuts down on heap traffic by statically allocating flat buffers large enough to hold maximums set by the application. It solves the two-pass proble…

They use it in android. They also use it in their firebase sdks for ios/android.

Nanopb doesnt solve wire issues. Thats for you to solve by designing your data better. Nanopb solves your applications binary size getting needlessly huge, due to less bloated code-gen and saner runtime (if you have to use protobuf, or are already married with it that is). It also gives more control over memory management, which is especially important for embedded.

Re: Don't Use Protobuf for Telemetry

#86

Post by tijsvd »

joshuamorton wrote: post #67
tijsvd wrote: post #63

Earlier quoted context omitted.

What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

[Also a googler]: PB is used for a wide variety of things, its a lingua-franca of data interchange within Google, but web (frontend) still often uses json. So if I want to have some proto in a database and render it on a UI to a user, somewhere in there I'm probably going to be translating the raw proto to either json or json-like (I'll admit I'm not sure how stuff like grpc-web fits in here, if you can just get prot…

Sure that makes some sense.

But it's somewhat ugly as well, from an architecture point of view. Because that argument translates to anything else you might want to do with these objects.

In past libraries for C++ I've tried to prevent this kind of coupling by adding generated template methods like "walk(f)" where f would be a templated callable, called with a descriptor and data reference for each field. Any kind of pretty printer or SQL statement can be built that way.

Re: Don't Use Protobuf for Telemetry

#87

Post by haberman »

seppel wrote: post #81
haberman wrote: post #61

Earlier quoted context omitted.

> Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages This is no longer the case if you use arenas: https://developers.google.com/protocol-buffers/docs/referenc... > and perhaps even for optional scalars This has never been the case, except for string fields where std::string forces us to allocate. Ideally we will eventually use std::string_view for st…

> This has never been the case, except for string fields where std::string forces us to allocate. I'm was quite surprised you didnt offer your own stringview implementation (or something similar) the last time I looked at protobuf. I'd naively assume that inside Google this could be quite a low-efford high-reward optimization.

> I'm was quite surprised you didnt offer your own stringview implementation (or something similar) the last time I looked at protobuf.

We sort of do actually: https://github.com/protocolbuffers/protobuf/blob/master/src/...

The internal version of protobuf lets you switch individual string fields to string_view using [ctype=STRING_PIECE], but migrating the default away from std::string is mainly just an enormous migration challenge.

Internally we also do something slightly nuts: we break the encapsulation of std::string so that we can point it to arena-allocated memory (we then "steal" the memory back before the destructor runs). We can only afford to do this internally, where the implementation of std::string is known. The real long-term solution is to move to string_view.

Re: Don't Use Protobuf for Telemetry

#88

Post by kentonv »

kentonv wrote: post #76
WorldMaker wrote: post #25

Earlier quoted context omitted.

It seems easy to square the practical experience of protobufs at Google versus the tradeoffs encountered in the wire format when you take into account how many Google engineers have directly contributed to the diaspora of related formats such as CapnProto, flatbuffers, msgpack, etc with different tradeoffs (especially for smaller scales than Google). It seems clear enough that protobufs were optimized in a scale that…

> It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully t…

> The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities.

And just to be clear, I don't think this is bad. On the contrary, I think Protobuf won because it did a wide variety of things "pretty well" while moving quickly and solving real problems. This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out something that works and running with it to solve real problems. If you try to carefully design everything perfectly upfront, you'll spend a huge amount of time on decisions that don't really matter.

Re: Don't Use Protobuf for Telemetry

#89

Post by kentonv »

haberman wrote: post #83
kentonv wrote: post #76

Earlier quoted context omitted.

> It seems clear enough that protobufs were optimized in a scale that involved a lot of time/cost-sensitive reads and far fewer time/cost-sensitive writes. Nah, you're assuming too much. Protobuf was thrown together in a fairly ad hoc way by a couple (brilliant!) engineers (Jeff and Sanjay) to help make the Google search index protocol easier to maintain. The specific design decisions in Protobuf were not carefully t…

Hey that reminds me, I've always wondered: how did it come to be that delimited encoding "won" for sub-messages and group encoding was deprecated? A lot of these problems would go away if things had gone the other way. Groups can be encoded in one pass, they are efficient to decode (unless you were trying to skip the sub-message, a la LazyField), and they don't have the string/message ambiguity in UnknownFieldSet tha…

That decision predates me. I think it was basically because early versions of protobuf didn't actually support using a message type as field type, so instead people would declare "string" fields and then manually encode/decode another protobuf type into that field. When the ability to explicitly use message types as field types was added to the language, they wanted to use it in those existing protocols without breaking compatibility, so the design was fit to the pre-existing practice.

I argued for switching to group encoding for submessages when working on proto2, but was shot down. It was a long time ago, but I think the counter-argument was some combination of "it's not worth the breakage" and "the ability to lazily parse sub-messages is too valuable".

Re: Don't Use Protobuf for Telemetry

#90

Post by sebastos »

kentonv wrote: post #52
alisonkisk wrote: post #48

Earlier quoted context omitted.

Part of the reason that Kenton left Google and created CapnProto, was because he was the only person working on proto at Google trying to fix the problems Google found.

That's not really accurate... I transferred off the Protobuf project three years before quitting Google, based on feedback I received from management suggesting they didn't think my work there was worthwhile. By the time I left Google, there was a new team maintaining Protobuf, and my reasons for leaving were not related to Protobuf. I didn't leave Google to create Cap'n Proto. Rather, having left Google and being fr…

Damned if you're not the most patient self-advocate for an open source project I've ever seen. I feel like I can always count on finding The Kenton Varda in the comments whenever a protobuf discussion comes up, graciously discussing design trade-offs and dispelling misconceptions.

Did you ever consider looking for official / commercial backing for Cap N Proto? It seems demonstrably better than all the alternatives, but I get the sense that it hasn't taken over because "nobody ever got fired for buying IBM". I know for certain I'd have an easier time pushing capnp in my own organization if it was getting that type of backing.

Post reply on HN