Don't Use Protobuf for Telemetry

Did you even read the post? It's obvious OP works at Datadog ( they mention it and mention the work is for their employer), so telemetry as in OpenTelemetry ( the new standard around metrics, traces and logs).

That's fair enough as a description of how the protocol was (not) designed that those sorts of trade-offs were not taken into account in the initial design.

However, it doesn't entirely invalidate my assumption: search index protocol is pretty obviously a context/scale that would have deeply favored more time/cost-sensitive reads (need to get search results to users as fast as possible) and the time/cost for writers less of a pressure (as writers for the index protocol could presumably be amortized with caching/proxying/throwing hardware at the problem). Whether that was an intentional "optimization" process or simply "optimization evolutionary pressure", it does seem to me (as an entirely outside observer) like a natural trade-off optimization that ocurred at Google scale for protobufs that kept protobufs feeling "well enough" that they were essentially left alone and never optimized for something with different trade-offs (such as use cases that were more write than read-heavy).

Which it is still useful to know those sorts of "optimization evolutionary pressures" to answer questions like "sure, it worked for Google, but will it work for me in this very different use case/scale?"

> Did you ever consider looking for official / commercial backing for Cap N Proto?

Well, I don't think Cap'n Proto is marketable (for any kind of profit) on its own. It's one of those things that has to be free and open source to get any adoption. So a corporate backer would need to get some indirect benefit from its development and adoption. Also, at this point my goal in building and maintaining Cap'n Proto is primarily to benefit my other projects that use it -- if other people benefit too, great, but wide adoption of Cap'n Proto is not an intrinsic goal of mine.

Right now, Cap'n Proto development is de facto backed by Cloudflare, as we are using it heavily in Cloudflare Workers and that is driving all the recent work on the C++ implementation. Currently, this use is internal-only and so we have little reason to build out support for multiple languages. But, as the Workers platform grows the ability to support increasingly complex apps and distributed systems (especially with Durable Objects), limiting applications to communicating with HTTP only is getting awkward. One idea that has been tossed around is to expose Cap'n Proto directly to apps. It makes a lot of sense: it would be very easy for us to support since we already use it under the hood, and zero-copy communications make a ton of sense especially for worker-to-worker comms happening within a single process. If we decided to do this, Cloudflare would become the commercial backer. But, there's also a strong argument to support gRPC directly in Workers, given the existing ecosystem. Would we want to support both? Maybe, maybe not. There's a lot of trade-offs still to think about here, and my goal would be to make the best possible product decision for Cloudflare Workers -- not necessarily for Cap'n Proto.

>Don't Use Protobuf for Telemetry

so I thought I had a shot of reading something relevant to me. So I opened up the article:

>Protobuf needs no introduction, but this post argues that you shouldn’t use it for telemetry. The basic premise of this post is that a good telemetry library needs to be lightweight to avoid perturbing the application; inefficient diagnostic tools are self-defeating. Unlike other formats, nested Protobuf messages cannot be written contiguously into a stream without significant buffering. The post doesn’t argue to never use Protobuf, but that the trade-off made by the wire-format itself, as opposed to any existing implementation, is unlikely to work for lightweight message senders.

My field is related to robotics, where we actually send, you know, telemetry. Like, over a radio. What I'm complaining about is that this is arguably the "main" usage of this word, but people in the web space assume that like protobuf, the concept of telemetry needs no introduction: it means sending back statistics, metrics and logs about your web services. When he says "a good telemetry library" needs to be lightweight, there's zero preamble here to get me oriented.

You saying "did you even read the post" is exactly missing the point. Yes, I read through the post, and through context clues, discerned that this post isn't relevant to my interests. I just notice that people writing about cloud service technology never feel compelled to warn you that's what they're going to be talking about. They make grand proclamations, and you have to play detective to find out whether their claims apply to you. "Microservices are the future!" "Okay, does he mean like... for all of software? Or for SaaS companies?"

Imagine if the world was a different place, and the average programmer worked at a robotics company instead of a web tech company. And then you saw an article titled "gRPC is dead - DDS is the future". And then it jumped in and started talking about all these throughput metrics, and it took you 6 paragraphs to figure out that the reason they're done with gRPC is that they're shipping around 4K images at 60fps to dozens of microservices that all run on the same computer. And you'd go oh, well that's... irrelevant. That's how it feels, all the time.