Don't Use Protobuf for Telemetry

Post by asdfasgasdgasdg » Thu, Dec 31, 2020, 1:15 AM UTC

rkagerer wrote: post #99

saberience wrote: post #97

Earlier quoted context omitted.

You don't work for Blizzard do you? When I worked there engineers (not my team thankfully) designed their own alternative to protobuf to try and save more bytes over the wire, which in my opinion was a really poor decision. Rather than get on with actually adding value, they ended up pushing back multiple other teams deadlines while adding almost no value. It was a classic "not built here" mentality and doing enginee…

If protobuf works for Google then it essentially works for 99.999% of every other company on the globe. I can't agree with this mindset. Another commenter here pointed out the Google implementation is 20x slower than others, and 1.6MB for this kind of task feels bloaty. Just because it meets Google's needs doesn't mean it's universally adequate.

Post by arcticbull » Thu, Dec 31, 2020, 1:18 AM UTC

jasonzemos wrote: post #107

haberman wrote: post #80

Earlier quoted context omitted.

> make apologies from authority (after all, Google can't do anything wrong, right?) That's not my position at all. In my other comment ( https://news.ycombinator.com/item?id=25586447 ) I explain how I've spent 10 years trying to improve on protobuf C++ precisely because I agree that some of these limitations are unnecessary. > No zero-copy for networking? Forced internal heap allocations with only this arena feature…

> I suppose it depends what you are comparing it to. Almost every JSON library has the same limitations you mentioned, and yet many people find JSON useful for network applications. JSON is a human-readable format which is hugely advantageous to develop and operate in many settings. Protobufs doesn't have that advantage. Yet we're paying all of the same costs to structure the data with both. That's enough to posit JS…

Post by jnwatson » Thu, Dec 31, 2020, 1:26 AM UTC

Post by viraptor » Thu, Dec 31, 2020, 1:31 AM UTC

sebastos wrote: post #100

sofixa wrote: post #94

Earlier quoted context omitted.

> Based on the average post you see on Hackernews, it's a virtual certainty he means "snooping on users of a program". I can't explain why, but it just bothers me that this would be considered so obvious as to escape mention. "Telemetry" means something, and it meant something to aerospace engineers for a long time before somebody at google started using it to refer to something incredibly narrow. Did you even read t…

Well, I started to read the article, thinking that it might refer to sending, you know, telemetry. The hackernews link was titled like this: >Don't Use Protobuf for Telemetry so I thought I had a shot of reading something relevant to me. So I opened up the article: >Protobuf needs no introduction, but this post argues that you shouldn’t use it for telemetry. The basic premise of this post is that a good telemetry lib…

Post by kentonv » Thu, Dec 31, 2020, 1:44 AM UTC

sebastos wrote: post #104

kentonv wrote: post #98

Earlier quoted context omitted.

Hah, thanks. > Did you ever consider looking for official / commercial backing for Cap N Proto? Well, I don't think Cap'n Proto is marketable (for any kind of profit) on its own. It's one of those things that has to be free and open source to get any adoption. So a corporate backer would need to get some indirect benefit from its development and adoption. Also, at this point my goal in building and maintaining Cap'n…

Thanks for the insight, that makes sense! Perhaps this is too unhip, but I wonder - have you ever dealt with the OMG's Data Distribution Service (DDS)? I used to think it was too fuddy-duddy to even glance at, but the recent adoption of DDS by ROS2 made me take another look. DDS seems to have a lot of interesting properties, particularly the QoS and discovery mechanisms. What made me think of it is that DDS is really…