Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenTelemetry: Escape Hatch from the Observability Cartel

oneuptime.com

Post reply on HN
Print view View original

41–50 of 73 posts

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#41

Post by ViewTrick1002 »

jitl wrote: post #37
ViewTrick1002 wrote: post #26

What is the preferred setup for deploying OpenTelemetry on a Kubernetes cluster? Is OpenTelemetry the choice today? I am running a few projects on a minimal Hetzner K3S cluster and just want some cheap easy observability to store logs, reduce log noise and instead rely on counters/metrics without paying an arm and a leg. Languages used are Rust, Javascript and Python mostly.

Otel collector or complete Otel stack with data storage and ui etc? The collector is a helm chart, someone on my project added it to our K8s clusters last week. It was like 30 lines of YAML/Terraform in total. Logs, trace forwarding, Prometheus scraping. That bit is easy. Idk about deploying the ui/storage. I’ve used Grafana Loki stuff in Docker Compose locally without much head scratching for local development. http…

Was thinking storage and UI. Making something useful of the work spent collecting the information!

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#42

Post by pas »

arccy wrote: post #39
pas wrote: post #30

Earlier quoted context omitted.

the problem is not simple either, and the same (or very similar) SDKs are used by the very fancy platforms too, no? compared to dumping logs to a file (or a single instance Prometheus scraping /metrics) everything is frustrating, because there are so many moving parts anyway, you want to query stuff and correlate, but for that you need to propagate the trace id, and emit and store spans, and take care to properly han…

nearly all the sdks are wild pieces of overengineering designed to conform to a design-by-committee api that's native to no language. half the things you list aren't even part of the sdks, they're part of the collector.

yes, and that's why it's usually better to put the spans where you actually need them instead of depending on monkey patching. still, it's the same whether you use DataDog or Sentry (or OTel).

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#43

Post by csomar »

kchoudhu wrote: post #15
andrewstuart2 wrote: post #7

Earlier quoted context omitted.

I'd just like to point out that you've said OTel has fundamental problems, and then you pointed out a couple examples of one-time-fixable transient problems. These are issues you'd experience with anything that spans your stack as a custom telemetry library would.

There is very much an alternative. Looking at the execution of your code should never alter its fundamental performance the way otel is built to do. This was a solved problem at least a decade and a half ago, but the cool kids decided to reinvent the wheel, poorly. https://news.ycombinator.com/item?id=45845889

That's just one dimension to telemetry. For my use case, for example, I need distributed tracing; which is a fancy word for correlated logs.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#44

Post by nothrabannosir »

PunchyHamster wrote: post #23
kchoudhu wrote: post #15

Earlier quoted context omitted.

There is very much an alternative. Looking at the execution of your code should never alter its fundamental performance the way otel is built to do. This was a solved problem at least a decade and a half ago, but the cool kids decided to reinvent the wheel, poorly. https://news.ycombinator.com/item?id=45845889

dtrace was meant for entirely different use, and it's not a replacement for otel Otel was made to basically track the request execution (and anything that request triggers) across multiple apps at once, not to instrument an app to find slow points

To OP’s credit though the latter is exactly what every single piece of otel documentation pushes you to do. Using only the manual spans api is an exercise in api docs spelunking and ignoring “suggested best practices” and “only do this if everything else has failed for you”.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#45

Post by kchoudhu »

PunchyHamster wrote: post #23
kchoudhu wrote: post #15

Earlier quoted context omitted.

There is very much an alternative. Looking at the execution of your code should never alter its fundamental performance the way otel is built to do. This was a solved problem at least a decade and a half ago, but the cool kids decided to reinvent the wheel, poorly. https://news.ycombinator.com/item?id=45845889

dtrace was meant for entirely different use, and it's not a replacement for otel Otel was made to basically track the request execution (and anything that request triggers) across multiple apps at once, not to instrument an app to find slow points

We should be using USDTs to emit trace ids that can be consumed by dtrace and shoved into whatever backend we want for tracing.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#46

Post by voxic11 »

jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

Have you looked at https://signoz.io/ ? Its the closest I have tried, but I still wouldn't put it quite at the same level of usability.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#47

Post by blyry »

nijave wrote: post #29
jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

I thought New Relic was pretty good but their per seat license cost is super high They have a SQL like query language that I think can do most of what you're describing

We migrated from newrelic to datadog (for cost reasons LMAO) a while back and I miss NRQL every single day I'm building a dashboard.

I enjoy having everything instrumented and in one spot, it's super powerful, but I am currently advocating for self hosting loki so that we can have debug+ level logs across all environments for a much much lower cost. Datadog is really good at identifying anomalies, but the cost for logs is so high there's a non-trivial amount of savings in sampling and minimizing logging. I HATE that we have told devs "don't log so much" -- that misses the entire point of building out a haystack. And sampling logs at 1%, and only logging warnings+ in prod makes it even harder to identify anomalies in lower environments before a prod release.

last hot take: The UX in kibana in 2016 was better than anything else we have now for rapidly searching through a big haystack, and identifying and correlating issues in logs.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#48

Post by tokol17982 »

jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

https://github.com/uptrace/uptrace OpenTelemetry-based, feels much closer to Datadog than Grafana.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#49

Post by masterj »

kchoudhu wrote: post #45
PunchyHamster wrote: post #23

Earlier quoted context omitted.

dtrace was meant for entirely different use, and it's not a replacement for otel Otel was made to basically track the request execution (and anything that request triggers) across multiple apps at once, not to instrument an app to find slow points

We should be using USDTs to emit trace ids that can be consumed by dtrace and shoved into whatever backend we want for tracing.

Why don’t you try that, convert the output to OTLP and then write about it?

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#50

Post by everfrustrated »

jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

As someone used to Datadog I can't find anything that comes close. Datadog makes it so easy to throw adhoc data up on a screen and explore a hypothesis from there. Using Grafana or New Relic feels like going from a 30" screen to a 14" laptop - everything is just harder and more frustrating in ways that are hard to articulate.
Post reply on HN