Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenTelemetry: Escape Hatch from the Observability Cartel

oneuptime.com

Post reply on HN
Print view View original

31–40 of 73 posts

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#31

Post by nijave »

ViewTrick1002 wrote: post #26

What is the preferred setup for deploying OpenTelemetry on a Kubernetes cluster? Is OpenTelemetry the choice today? I am running a few projects on a minimal Hetzner K3S cluster and just want some cheap easy observability to store logs, reduce log noise and instead rely on counters/metrics without paying an arm and a leg. Languages used are Rust, Javascript and Python mostly.

Not sure about preferred but Jaeger is pretty simple for tracing only. Hyperdx looks interesting but brings in both Mongo and Clickhouse which imo is pretty heavy

Native k8s logs + Prometheus is probably more on the lighter weight side but you don't get traces. You could find some middle ground using the otel collector to extract trace metrics so you get RED metrics but you wouldn't have full traces

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#32

Post by nijave »

ramon156 wrote: post #24

OTel has been slow as hell since release. we tried using it for tracing last week and the app was significantly slower. We just use prometheus+grafana now. Yes, this technically also slows the app down, but OTel was unbearably slow. I'm sure im doing a million and one things wrong, but i can't be arsed to set something up just to see some performance metrics. Deadlocks can be found using transaction metrics, that's a…

You might be able to use big telemetry vendor libraries for instrumentation. Iirc you can use ddtrace Python library and have it send to otel collector which will convert from DD span format to otel.

I haven't tried but it's probably possible to do the same with JS

I think Sentry was also expanding into tracing--might be worth a look to see if they're doing something that works better in their library

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#33

Post by nijave »

I guess I'm the contrarian but I've had good success with otel. It's especially powerful being able to plug and play components like take ddtrace Python instrumentation library and hook it up to otel collector and dump into Jaeger.

I can run a full tracing stack locally for dev use with minimal config.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#35

Post by jitl »

nijave wrote: post #29
jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

I thought New Relic was pretty good but their per seat license cost is super high They have a SQL like query language that I think can do most of what you're describing

I want improvements like “dropdown menu of tags not limited to 50 items” and “one button forecast on any graph”. Unsure if any SQL query language will solve my annoyance that stuff in Datadog was one click, and in Grafana is 14 clicks or impossible without referencing a Book of the Arcane.

Heck I’m dying for “I can copy a graph between dashboards”. Grafana allows this but if any variable is in the graph but doesn’t exist in the destination, pasting just creates an empty graph.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#36

Post by secondcoming »

mohammadv184 wrote: post #27

There is an un-marketed reality: OTel is not simple. The learning curve is steep, the documentation can be a maze of specs, and the SDKs (especially for metrics and logs) can feel over-engineered. You will get frustrated.

The C++ SDK is a masterpiece of over-engineering.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#37

Post by jitl »

ViewTrick1002 wrote: post #26

What is the preferred setup for deploying OpenTelemetry on a Kubernetes cluster? Is OpenTelemetry the choice today? I am running a few projects on a minimal Hetzner K3S cluster and just want some cheap easy observability to store logs, reduce log noise and instead rely on counters/metrics without paying an arm and a leg. Languages used are Rust, Javascript and Python mostly.

Otel collector or complete Otel stack with data storage and ui etc?

The collector is a helm chart, someone on my project added it to our K8s clusters last week. It was like 30 lines of YAML/Terraform in total. Logs, trace forwarding, Prometheus scraping. That bit is easy.

Idk about deploying the ui/storage. I’ve used Grafana Loki stuff in Docker Compose locally without much head scratching for local development.

https://github.com/grafana/docker-otel-lgtm

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#38

Post by arccy »

jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

really? for a lot of us the datadog query language for dashboards is absolute trash. getting it to do what you want sits somewhere between difficult to impossible.

i guess it depends on what you're used to

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#39

Post by arccy »

pas wrote: post #30
mohammadv184 wrote: post #27

There is an un-marketed reality: OTel is not simple. The learning curve is steep, the documentation can be a maze of specs, and the SDKs (especially for metrics and logs) can feel over-engineered. You will get frustrated.

the problem is not simple either, and the same (or very similar) SDKs are used by the very fancy platforms too, no? compared to dumping logs to a file (or a single instance Prometheus scraping /metrics) everything is frustrating, because there are so many moving parts anyway, you want to query stuff and correlate, but for that you need to propagate the trace id, and emit and store spans, and take care to properly han…

nearly all the sdks are wild pieces of overengineering designed to conform to a design-by-committee api that's native to no language.

half the things you list aren't even part of the sdks, they're part of the collector.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#40

Post by mvf4z7 »

jitl wrote: post #21
greatgib wrote: post #16

I was always turned down to use more Otel by how verbose it is and how heavy are the telemetry payload compared to simple adhoc alternatives. Am I wrong?

The JavaScript Otel packages are implemented with 34 layers of extra abstraction. We wrote our own implementation of tracing for Cloudflare Workers and it performs much better with 0 layers of abstraction. I’ve seen a few other services switching over to our lightweight tracer. The emitted JSON is still chunky but removing all the incidental complexity helped a lot.

Can you share a link to your implementation?
Post reply on HN