OpenTelemetry: Escape Hatch from the Observability Cartel • Hacker News Board

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#21

Post by jitl » Fri, Nov 07, 2025, 1:11 PM UTC

greatgib wrote: post #16

I was always turned down to use more Otel by how verbose it is and how heavy are the telemetry payload compared to simple adhoc alternatives. Am I wrong?

The JavaScript Otel packages are implemented with 34 layers of extra abstraction. We wrote our own implementation of tracing for Cloudflare Workers and it performs much better with 0 layers of abstraction. I’ve seen a few other services switching over to our lightweight tracer. The emitted JSON is still chunky but removing all the incidental complexity helped a lot.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#22

Post by PunchyHamster » Fri, Nov 07, 2025, 1:12 PM UTC

csomar wrote: post #2

What the author doesn't realize is that OpenTelemetry has fundamental problems. I experienced this firsthand two years ago working with Otel in Rust, and just today I spent an entire afternoon debugging what turned out to be an otel package update breaking react-router links. Since the bug showed up with several other package updates at once, otel was in the bottom of my suspicion list. The core issue is that, with o…

Most languages have pretty mature ecosystem, I used it in Go and it was mostly problem free, with biggest annoyance being a bit of boilerplate that had to be added

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#23

Post by PunchyHamster » Fri, Nov 07, 2025, 1:14 PM UTC

kchoudhu wrote: post #15

andrewstuart2 wrote: post #7

Earlier quoted context omitted.

I'd just like to point out that you've said OTel has fundamental problems, and then you pointed out a couple examples of one-time-fixable transient problems. These are issues you'd experience with anything that spans your stack as a custom telemetry library would.

There is very much an alternative. Looking at the execution of your code should never alter its fundamental performance the way otel is built to do. This was a solved problem at least a decade and a half ago, but the cool kids decided to reinvent the wheel, poorly. https://news.ycombinator.com/item?id=45845889

dtrace was meant for entirely different use, and it's not a replacement for otel

Otel was made to basically track the request execution (and anything that request triggers) across multiple apps at once, not to instrument an app to find slow points

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#24

Post by ramon156 » Fri, Nov 07, 2025, 1:15 PM UTC

OTel has been slow as hell since release. we tried using it for tracing last week and the app was significantly slower.

We just use prometheus+grafana now. Yes, this technically also slows the app down, but OTel was unbearably slow.

I'm sure im doing a million and one things wrong, but i can't be arsed to set something up just to see some performance metrics. Deadlocks can be found using transaction metrics, that's all you need.

Edit: I now read in the comments the JS ver is a bad impl, I guess this might be part of the reason.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#25

Post by jitl » Fri, Nov 07, 2025, 1:24 PM UTC

MrBuddyCasino wrote: post #20

jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

dash0 is a brand new player, and wants to be "simple", perhaps check it out. Former colleagues of mine so I'm biased, but they do know what they're doing - they built the APM tool Instana, which sold to IBM for $400M.

You can make a tool that’s both powerful and usable. Datadog did it. Adobe did it. There is no downside to usability.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#26

Post by ViewTrick1002 » Fri, Nov 07, 2025, 1:24 PM UTC

What is the preferred setup for deploying OpenTelemetry on a Kubernetes cluster? Is OpenTelemetry the choice today?

I am running a few projects on a minimal Hetzner K3S cluster and just want some cheap easy observability to store logs, reduce log noise and instead rely on counters/metrics without paying an arm and a leg.

Languages used are Rust, Javascript and Python mostly.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#27

Post by mohammadv184 » Fri, Nov 07, 2025, 1:26 PM UTC

There is an un-marketed reality: OTel is not simple. The learning curve is steep, the documentation can be a maze of specs, and the SDKs (especially for metrics and logs) can feel over-engineered. You will get frustrated.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#28

Post by rixed » Fri, Nov 07, 2025, 1:30 PM UTC

kchoudhu wrote: post #13

OpenTelemetry won observability mindshare, but it is entirely the wrong architectural choice: by buying into its ethos your code is held hostage by the least stable otel monitoring library for your dependencies. Sadly, there was always an alternative that no one took: dtrace. Add USDTs to your code and then monitor progress by instrumenting it externally, sending the resulting traces to wherever you want. My sincere…

I'm not familiar with usdt, but my understanding is that it covers only the lower level of Opentelemetry. Is there an equivalent of OTLP for dtrace?

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#29

Post by nijave » Fri, Nov 07, 2025, 1:31 PM UTC

jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

I thought New Relic was pretty good but their per seat license cost is super high

They have a SQL like query language that I think can do most of what you're describing

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#30

Post by pas » Fri, Nov 07, 2025, 1:35 PM UTC

mohammadv184 wrote: post #27

There is an un-marketed reality: OTel is not simple. The learning curve is steep, the documentation can be a maze of specs, and the SDKs (especially for metrics and logs) can feel over-engineered. You will get frustrated.

the problem is not simple either, and the same (or very similar) SDKs are used by the very fancy platforms too, no?

compared to dumping logs to a file (or a single instance Prometheus scraping /metrics) everything is frustrating, because there are so many moving parts anyway, you want to query stuff and correlate, but for that you need to propagate the trace id, and emit and store spans, and take care to properly handle async workers, and you want historical comparisons, and downsampling for retention, and of course auto-discovery/registration/labeling from k8s pods (or containers or whatever) and source code upload and release tagging from CI, and ...