Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't Use Protobuf for Telemetry

richardstartin.github.io

Post reply on HN
Print view View original

61–70 of 195 posts

Re: Don't Use Protobuf for Telemetry

#61

Post by haberman »

tijsvd wrote: post #54
thisiscorrect wrote: post #34

Earlier quoted context omitted.

Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated? [edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?

Things may have improved since, but the implementations are somehow very large and slow. Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages, and perhaps even for optional scalars. This may be optimal for larger documents, but not for smallish messages (my use case was market data and trading instructions). I measured certain small messages, where an e…

> Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages

This is no longer the case if you use arenas: https://developers.google.com/protocol-buffers/docs/referenc...

> and perhaps even for optional scalars

This has never been the case, except for string fields where std::string forces us to allocate.

Ideally we will eventually use std::string_view for string accessors instead of std::string, so that even string data can be allocated on an arena instead of the heap.

Re: Don't Use Protobuf for Telemetry

#62

Post by kentonv »

tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered).

Right, they were designed for use in Google's servers, where binary size is mostly irrelevant, while speed and features (e.g. reflection) are valued.

"Lite mode" (not mentioned in the article, for some reason) optimizes for code size instead. Admittedly, it's not as small as an implementation written from scratch with minimal code footprint as the main goal, but it does cut the size quite a bit...

(Disclaimer: I wrote those C++ and Java implementations, but that was 10-15 years ago, things may have changed in the meantime...)

Re: Don't Use Protobuf for Telemetry

#63

Post by tijsvd »

haberman wrote: post #53
tijsvd wrote: post #31

I have written an in-house implementation of protobuf for C++ (sorry can't share) and studied the wire format extensively. Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Don't shoot down the format because of a specific implementation, find or write a better one and enjoy the fact that every language has at…

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). As someone who has been working on protobuf-related things for >10 years, including creating a size-focused implementation ( https://github.com/protocolbuffers/upb ), and has been working on the protobuf team for >5 years, I have a few thoughts on this (though…

What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

Re: Don't Use Protobuf for Telemetry

#64

Post by kentonv »

tijsvd wrote: post #54
thisiscorrect wrote: post #34

Earlier quoted context omitted.

Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated? [edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?

Things may have improved since, but the implementations are somehow very large and slow. Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages, and perhaps even for optional scalars. This may be optimal for larger documents, but not for smallish messages (my use case was market data and trading instructions). I measured certain small messages, where an e…

> AFAIK the C++ implementation would always allocate on the heap for nested messages

FWIW if you reuse the same message object for multiple parsings, it will re-use the sub-objects as well, thus amortizing away the allocation cost. Parsing the same message into the same object twice should do zero allocations on the second parse. This is the intended way to use Protobuf for small-size messages.

Apparently the C++ implementation has also grown support for arena allocation more recently. (After my time, so I don't know much about it.)

Re: Don't Use Protobuf for Telemetry

#65

Post by Aaargh20318 »

Nvorzula wrote: post #39
Aaargh20318 wrote: post #19

Are there any advantages of protobuf over using DER encoded ASN.1 ? The format looks very similar. It looks like they kind of reinvented the wheel here.

To my understanding they're both [type, length, content] encoded formats, so they're similar there at least. In my experience people really quite enjoy the IDL aspect of protobuf as the killer feature, and I am not sure any such compiler exists for ASN1.

ASN.1 is basically an IDL. There are multiple encodings for ASN.1 but DER is one of the most common ones (you can also encode to JSON or XML).

DER is indeed TLV (tag-length-value) and there are many code generators available.

Re: Don't Use Protobuf for Telemetry

#66

Post by jasonzemos »

tijsvd wrote: post #54
thisiscorrect wrote: post #34

Earlier quoted context omitted.

Mind elaborating on how the "standard" C++ and Java protobuf implementations are bloated? [edit] I'm genuinely asking. I'm guessing you mean in the generated APIs, i.e. the code "weight", but maybe you meant something else?

Things may have improved since, but the implementations are somehow very large and slow. Things may have changed since, but AFAIK the C++ implementation would always allocate on the heap for nested messages, and perhaps even for optional scalars. This may be optimal for larger documents, but not for smallish messages (my use case was market data and trading instructions). I measured certain small messages, where an e…

Since all other comments appear to contradict you and make apologies from authority (after all, Google can't do anything wrong, right?) I'd like to just reassure you with my 20 years of experience developing C (and the last 10 with C++) in network and system software: that any library -- any library -- that forces internal dynamic memory upon its user smells bad. It's not a universal condemnation, but it begs the question, or rather the skepticism to ask why?

In the case of protobufs, there is no real answer. Protobufs is one of the few network libraries that doesn't operate with zero-copy. That steps over a tangible line in the sand.

No zero-copy for networking? Forced internal heap allocations with only this arena feature after a decade? Sorry no. Protobufs isn't useful for serious network applications.

Re: Don't Use Protobuf for Telemetry

#67

Post by joshuamorton »

tijsvd wrote: post #63
haberman wrote: post #53

Earlier quoted context omitted.

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). As someone who has been working on protobuf-related things for >10 years, including creating a size-focused implementation ( https://github.com/protocolbuffers/upb ), and has been working on the protobuf team for >5 years, I have a few thoughts on this (though…

What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

[Also a googler]:

PB is used for a wide variety of things, its a lingua-franca of data interchange within Google, but web (frontend) still often uses json. So if I want to have some proto in a database and render it on a UI to a user, somewhere in there I'm probably going to be translating the raw proto to either json or json-like (I'll admit I'm not sure how stuff like grpc-web fits in here, if you can just get proto-serialized bytes passed around in some a blob, but there are some pretty low-overhead json proto encodings that exist anyway).

Re: Don't Use Protobuf for Telemetry

#68

Post by kentonv »

tijsvd wrote: post #63
haberman wrote: post #53

Earlier quoted context omitted.

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). As someone who has been working on protobuf-related things for >10 years, including creating a size-focused implementation ( https://github.com/protocolbuffers/upb ), and has been working on the protobuf team for >5 years, I have a few thoughts on this (though…

What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

People implementing public-facing APIs might not be comfortable forcing all the consumers of that API to use Protobuf, but may want to use Protobuf internally within their service. So, Protobuf lets you convert to/from JSON, so that your service can expose a JSON API while maintaining Protobuf schemas and never dealing with a JSON parser directly.

The implementation is based on Protobuf's existing reflection features, so if you are already compiling with that enabled, the JSON implementation does not add much code bloat on top of that.

Re: Don't Use Protobuf for Telemetry

#69

Post by mianos »

Nvorzula wrote: post #39
Aaargh20318 wrote: post #19

Are there any advantages of protobuf over using DER encoded ASN.1 ? The format looks very similar. It looks like they kind of reinvented the wheel here.

To my understanding they're both [type, length, content] encoded formats, so they're similar there at least. In my experience people really quite enjoy the IDL aspect of protobuf as the killer feature, and I am not sure any such compiler exists for ASN1.

ASN.1 is specified as a compilable language. Open source compilers, such as snacc, have existed for 30 years. ASN.1 also defines multiple encodings, DER is used in crypto as you can compare encoded blocks bit by bit. Other encoding rules, such as BER, are tagged. Protobuffers is the same thing re-invented by people who probably never saw ASN.1.

Re: Don't Use Protobuf for Telemetry

#70

Post by brown9-2 »

Animats wrote: post #46
haberman wrote: post #44

Earlier quoted context omitted.

The author of this piece is wanting to stream possibly multi-MB messages with sub-linear memory use. See more in our Twitter conversation here: https://twitter.com/richardstartin/status/134406813297822105...

"possibly multi-MB messages" Megabytes? By "telemetry" does he mean "getting info on what our pumps out in the oil field are doing" or "snooping on users of a program"?

think of the type of monitoring/observability data that Datadog's agent collects
Post reply on HN