Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't Use Protobuf for Telemetry

richardstartin.github.io

Post reply on HN
Print view View original

151–160 of 195 posts

Re: Don't Use Protobuf for Telemetry

#151

Post by kentonv »

0xbadcafebee wrote: post #133
kentonv wrote: post #88

Earlier quoted context omitted.

> The specific design decisions in Protobuf were not carefully tested or weighed against other possibilities. And just to be clear, I don't think this is bad. On the contrary, I think Protobuf won because it did a wide variety of things "pretty well" while moving quickly and solving real problems. This is how the best technologies are usually made, not by academically trying to perfect everything, but by banging out…

If you're designing a critical component that's going to impact your business indefinitely, you can take some time for due diligence. This is basic Shift Left mentality and it has lots of benefits.

This was 2001-ish. No one at the time had any idea how widely the thing would end up being used, nor how big the company would grow. And spending too much time dwelling on each part of the tech stack then could easily have given a competitor the opportunity to pull ahead, in which case Google wouldn't be what it is today.

Re: Don't Use Protobuf for Telemetry

#152

Post by kentonv »

girvo wrote: post #142
kentonv wrote: post #62

Earlier quoted context omitted.

> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). Right, they were designed for use in Google's servers, where binary size is mostly irrelevant, while speed and features (e.g. reflection) are valued. "Lite mode" (not mentioned in the article, for some reason) optimizes for code size instead. Admittedly, it's…

HN is one of the few places you can talk smack about a companies tooling and have the person who wrote it reply! Lite mode is neat, I hadn’t seen that. That’s super useful for my use case, cheers. Any gotchas with it to be aware of?

> Any gotchas with it to be aware of?

Lots, unfortunately. E.g. you can't stringify objects for debug printing, because stringification is implemented in terms of reflection.

Re: Don't Use Protobuf for Telemetry

#153

Post by [deleted] »

jasonzemos wrote: post #149
kentonv wrote: post #141

Earlier quoted context omitted.

If you are arguing that there exist use cases for which Protobuf is ill-suited, then fine, I obviously agree. But you seem to be arguing that Protobuf is bad because it is not well suited to certain use cases, dismissing all other use cases as "non-serious". That is offensive.

I asked you not to argue semantics, but you're doing it anyway. I didn't respond here to argue protobufs actually is bad, since all software is suited to something for somebody somewhere. In fact, I can just do that now: Protobufs is bad. You seem to agree, because you haven't even advocated for it -- just flatbuffers and Cap'n Proto and the phonebook of alternatives. Listen, I don't care who you've worked for or wha…

[deleted]

Re: Don't Use Protobuf for Telemetry

#154

Post by 1f60c »

haberman wrote: post #74
tijsvd wrote: post #63

Earlier quoted context omitted.

What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

> Don't people use pb so they do not have to deal with JSON? I think that's one reason, but there are others too, like getting generated type-safe accessors and an explicit schema. I've sometimes used protobuf for this reason even when I'm not planning to use binary format. > I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size? Here is a breakdown of a binary that uses upb…

How'd you generate that listing? It seems useful.

Re: Don't Use Protobuf for Telemetry

#155

Post by mcguire »

kentonv wrote: post #102
jasonzemos wrote: post #66

Earlier quoted context omitted.

Since all other comments appear to contradict you and make apologies from authority (after all, Google can't do anything wrong, right?) I'd like to just reassure you with my 20 years of experience developing C (and the last 10 with C++) in network and system software: that any library -- any library -- that forces internal dynamic memory upon its user smells bad. It's not a universal condemnation, but it begs the que…

Unless you are targeting a constrained embedded system, this kind of anti-memory-allocation thinking is counter-productive. Avoiding memory allocation significantly increases the complexity of an API -- or worse, leads to shortcuts like using (thread-unsafe) globals or (overrun-prone) fixed buffers. In C, dealing with memory allocation is such a pain that C programmers still tend to avoid it. But C++, especially post…

Once upon a time, we were concerned with handling network traffic at wire speed. I can remember when the AIX folks were happy they could implement the TCP fast path entirely in the interrupt handler.

Re: Don't Use Protobuf for Telemetry

#157

Post by Animats »

sebastos wrote: post #100
sofixa wrote: post #94

Earlier quoted context omitted.

> Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow. Did you even read t…

Well, I started to read the article, thinking that it might refer to sending, you know, telemetry. The hackernews link was titled like this: >Don't Use Protobuf for Telemetry so I thought I had a shot of reading something relevant to me. So I opened up the article: >Protobuf needs no introduction, but this post argues that you shouldn’t use it for telemetry. The basic premise of this post is that a good telemetry lib…

Yes. Just recently I discovered that some web types refer to generating HTML from some other representation as "rendering". To graphics people, and artists, rendering is taking a description of something and turning it into a directly viewable picture. HTML is a long, long way from displayed pixels.

Come on. There's a big industry out there working to push game and entertainment content through the GPU and onto the screen. That's rendering.

Re: Don't Use Protobuf for Telemetry

#158

Post by VikingCoder »

sebastos wrote: post #100
sofixa wrote: post #94

Earlier quoted context omitted.

> Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow. Did you even read t…

Well, I started to read the article, thinking that it might refer to sending, you know, telemetry. The hackernews link was titled like this: >Don't Use Protobuf for Telemetry so I thought I had a shot of reading something relevant to me. So I opened up the article: >Protobuf needs no introduction, but this post argues that you shouldn’t use it for telemetry. The basic premise of this post is that a good telemetry lib…

Software is now literally everywhere, doing everything.

Picture if there was a website dedicated to Math News. Some articles talk about how amazing Tangents are. Some talk about Bayesian inference. It would be a mess.

Yes, Hacker News is a mess. That's pretty unavoidable. All conversations take place within a domain - a culture, a set of shared assumptions. And if you start talking with someone new, yes, it will take you quite a while to find out what assumptions they have that are different from yours.

Like, I wander in to an economics class, and they put the independent axis in the wrong direction.

I open up a computer graphics library, and it's using the left-hand rule.

One textbook says negative charge goes this way, and the textbook for a different course says it goes the other way.

Yes, people don't even understand that others MIGHT not understand their domain.

Yes, that is the world.

Re: Don't Use Protobuf for Telemetry

#159

Post by mintyc »

kentonv wrote: post #130
mintyc wrote: post #126

Earlier quoted context omitted.

Cap'n'proto has a number of drawbacks both from the 'one man band' pov. (Witness the two year lull when Kenton went sandstorming), an awkward API and limited language support. Personally I don't think there is a perfect protocol because different people want different things whether self describing, easy/optimised memory management, zero copy, partial decode. The list goes on.... At a pinch, flatbuffers with flexbuff…

> Witness the two year lull when Kenton went sandstorming There wasn't really a lull; development on Cap'n Proto continued that whole time by the Sandstorm team in service of Sandstorm. There was an absence of official releases since Sandstorm always used the latest Cap'n Proto code from git. The same story continues today with Sandstorm replaced by Cloudflare Workers. TBH I should probably give up on "official relea…

Wasn't a criticism, just an observation from a 'user' when a key bit of infrastructure is directed by a single or small team.

You are opening up a technology that you are crafting for your own purposes in the best spirit of sharing.

Personally I'm wary of relying on any project driven in this way.

Inspiring, but not something I'm personally keen on using directly.

I do feel grateful that you've given this as an option and I do feel slightly parasitic in not providing a tangible positive contribution instead.

Re: Don't Use Protobuf for Telemetry

#160

Post by haberman »

1f60c wrote: post #154
haberman wrote: post #74

Earlier quoted context omitted.

> Don't people use pb so they do not have to deal with JSON? I think that's one reason, but there are others too, like getting generated type-safe accessors and an explicit schema. I've sometimes used protobuf for this reason even when I'm not planning to use binary format. > I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size? Here is a breakdown of a binary that uses upb…

How'd you generate that listing? It seems useful.

I used my tool Bloaty, which parses the ELF and DWARF information in the binary: https://github.com/google/bloaty
Post reply on HN