Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenTelemetry: Escape Hatch from the Observability Cartel

oneuptime.com

Post reply on HN
Print view View original

11–20 of 73 posts

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#11

Post by andrewstuart2 »

csomar wrote: post #9
andrewstuart2 wrote: post #7

Earlier quoted context omitted.

I'd just like to point out that you've said OTel has fundamental problems, and then you pointed out a couple examples of one-time-fixable transient problems. These are issues you'd experience with anything that spans your stack as a custom telemetry library would.

It's more than a couple. The fundamental issue is not the bugs themselves (these are expected) but that, from my perspective, otel is at odds with the observability business because these actors have little interest to contribute back to telemetry agents since anyone can reap the rewards of that. So instead they'd focus more on their platforms and the agents/libraries get neglected. It's a great idea, in principle, b…

That's kind of how open source works, though. Of course the backend vendors won't care about anything that doesn't affect the backend somehow. But the people, i.e. users, who do want to be able to easily switch away from bad vendors, have incentives to keep things properly decoupled and working.

The license is the key enabler for all of this. The vendors can't be all that sneaky in the code they contribute without much higher risk of being caught. Sure, they will focus on the funnel that brings more data to them, but that leaves others more time to work on the other parts.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#12

Post by zja »

csomar wrote: post #9
andrewstuart2 wrote: post #7

Earlier quoted context omitted.

I'd just like to point out that you've said OTel has fundamental problems, and then you pointed out a couple examples of one-time-fixable transient problems. These are issues you'd experience with anything that spans your stack as a custom telemetry library would.

It's more than a couple. The fundamental issue is not the bugs themselves (these are expected) but that, from my perspective, otel is at odds with the observability business because these actors have little interest to contribute back to telemetry agents since anyone can reap the rewards of that. So instead they'd focus more on their platforms and the agents/libraries get neglected. It's a great idea, in principle, b…

> otel is at odds with the observability business because these actors have little interest to contribute back to telemetry agents since anyone can reap the rewards of that.

But all major vendors _do_ contribute to OTEL.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#13

Post by kchoudhu »

OpenTelemetry won observability mindshare, but it is entirely the wrong architectural choice: by buying into its ethos your code is held hostage by the least stable otel monitoring library for your dependencies.

Sadly, there was always an alternative that no one took: dtrace. Add USDTs to your code and then monitor progress by instrumenting it externally, sending the resulting traces to wherever you want. My sincere hope is that the renewed interest in ebpf makes this a reality soon: I never want to have to do another from opentelemetry. import ever again.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#14

Post by jitl »

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog?

Honeycomb is decent at what it has, but very limited offerings for dashboard.

Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation, even time shifting is like pulling teeth. Plus they just totally broke all our graphs with formulas for like 2 days.

Grafana is to Datadog what Bugzilla is to Linear.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#15

Post by kchoudhu »

andrewstuart2 wrote: post #7
csomar wrote: post #2

What the author doesn't realize is that OpenTelemetry has fundamental problems. I experienced this firsthand two years ago working with Otel in Rust, and just today I spent an entire afternoon debugging what turned out to be an otel package update breaking react-router links. Since the bug showed up with several other package updates at once, otel was in the bottom of my suspicion list. The core issue is that, with o…

I'd just like to point out that you've said OTel has fundamental problems, and then you pointed out a couple examples of one-time-fixable transient problems. These are issues you'd experience with anything that spans your stack as a custom telemetry library would.

There is very much an alternative. Looking at the execution of your code should never alter its fundamental performance the way otel is built to do. This was a solved problem at least a decade and a half ago, but the cool kids decided to reinvent the wheel, poorly.

https://news.ycombinator.com/item?id=45845889

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#17

Post by MrBuddyCasino »

greatgib wrote: post #16

I was always turned down to use more Otel by how verbose it is and how heavy are the telemetry payload compared to simple adhoc alternatives. Am I wrong?

No, the spec isn't great and makes it hard to implement a performant solution.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#18

Post by jitl »

kchoudhu wrote: post #13

OpenTelemetry won observability mindshare, but it is entirely the wrong architectural choice: by buying into its ethos your code is held hostage by the least stable otel monitoring library for your dependencies. Sadly, there was always an alternative that no one took: dtrace. Add USDTs to your code and then monitor progress by instrumenting it externally, sending the resulting traces to wherever you want. My sincere…

If some tracing plug in is shitting up your code with its monkeypatching, rip it out and instrument it yourself. We do this a lot. I’d say Otel packages are no better or worse quality wise than any other stuff in node_modules. Not otel’s fault that Code in general has Bugs and is Bad.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#19

Post by PunchyHamster »

greatgib wrote: post #16

I was always turned down to use more Otel by how verbose it is and how heavy are the telemetry payload compared to simple adhoc alternatives. Am I wrong?

It is designed to get insights into dozen(s) connected systems.

It will always be overkill for just an app or two talking with eachother... till you grow, then it won't be overkill any more.

But still might be worth getting into it on smaller apps just thanks to wealth of tools available.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#20

Post by MrBuddyCasino »

jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

dash0 is a brand new player, and wants to be "simple", perhaps check it out. Former colleagues of mine so I'm biased, but they do know what they're doing - they built the APM tool Instana, which sold to IBM for $400M.
Post reply on HN