Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OTel isn’t going well

matduggan.com

Post reply on HN
Print view View original

21–30 of 123 posts

Re: OTel isn’t going well

#21

Post by masterj »

cyberax wrote: post #13

I disagree. I'm an observability geek, and OTel is... fine. It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements: 1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" quer…

HN always grumbles about OTel, but I agree. It's fine, and important: https://jeremymorrell.dev/blog/opentelemetry-and-the-value-o...

I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects

Re: OTel isn’t going well

#22

Post by arcanemachiner »

cyberax wrote: post #14
GauntletWizard wrote: post #5

Earlier quoted context omitted.

The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone . This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither. Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated , either, and in fact the fact that they're not c…

OTEL metrics are a bit awkward, but they work just fine with Prometheus. Jaeger uses the OTLP protocol nowadays. So it _is_ OTEL.

What, so people don't like OTel, but they like Jaeger, which implements an OTel spec? (I'm a noob to this subject, if that wasn't obvious.)

Re: OTel isn’t going well

#23

Post by rcleveng »

Sounds a lot like K8s. It's not a framework you use, it's a framework to build a framework on top of.

I wish the observability vendors would move to using it under the covers so it's easier to mix and match.

I wish the otel support wasn't super buggy in most of the frameworks and backends.

Re: OTel isn’t going well

#25

Post by ffsm8 »

EdSchouten wrote: post #12
thorian1828i03 wrote: post #10

Earlier quoted context omitted.

One strategy do to do that is to trace everything by default and select what to sample later, e.g. https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...

If I understand that correctly, it means your app always creates traces, and Grafana Cloud is responsible for sampling/aggregating. That may be prohibitively expensive in terms of CPU/network load. What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries. The same thing with metri…

You're pitching a solution that's incredible brittle and unnecessarily complicated if you think about it in technical terms.

For your feature to work you need bi-directional communication between the otel receiver and your application - that's still doable in general, but now you want a synchronous "upgrade" to traces.

Now we're talking about a massive performance impact - and you need to somehow cache all otel data locally so they're available for the upgrade and only then submit then.

It is a architecture that's not very smart, honestly. And precisely the reason why you'd simply submit everything and let the receiver figure out which samples it wants to keep - as thorian pointed out earlier.

Re: OTel isn’t going well

#26

Post by bilalq »

cyberax wrote: post #19
bilalq wrote: post #7

OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today: 1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time. 2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now. 3. Serverles…

> 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage. You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent. App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the UR…

Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads.

Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to.

In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and then have a log stream processor that then writes to your gateway collector. Other than that, it seems unavoidable, no? Sure, in something like Fargate you could go app to sink. But even that has its own tradeoffs.

Re: OTel isn’t going well

#27

Post by bilalq »

arcanemachiner wrote: post #20
bilalq wrote: post #7

OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today: 1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time. 2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now. 3. Serverles…

So what's the alternative then? (Genuine question, not hypothetical snark.)

There isn't really a great alternative without vendor lock-in. If you go all-in on AWS Cloudwatch/X-Ray, it's a really easy setup with low effort. If you go all-in on Datadog, it's pretty easy. But if you want to mix Sentry, Langfuse, Datadog, etc, OTel is still probably the best option. It's just a letdown that this is the best there is.

I don't mean to disparage anyone working on OTel. I can appreciate that it has ambitious goals and it's not an easy problem to get alignment and interop here. Especially with all the stakeholders involved. But as a user, it feels simultaeneously over-engineered and under-engineered.

Re: OTel isn’t going well

#28

Post by steerpike »

masterj wrote: post #21
cyberax wrote: post #13

I disagree. I'm an observability geek, and OTel is... fine. It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements: 1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" quer…

HN always grumbles about OTel, but I agree. It's fine, and important: https://jeremymorrell.dev/blog/opentelemetry-and-the-value-o... I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects

Oh my god. A Jeremy Morrell sighting in the wild.

Every time I share your blog (and I share it a lot) I tell people:

"This guy started a blog in 2024. Wrote three posts and all three of them would still make my top ten list of 'greatest posts on observability' today".

'A practitioner's guide to wide events' especially is still my number 1.

Re: OTel isn’t going well

#29

Post by ragall »

EdSchouten wrote: post #12
thorian1828i03 wrote: post #10

Earlier quoted context omitted.

One strategy do to do that is to trace everything by default and select what to sample later, e.g. https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...

If I understand that correctly, it means your app always creates traces, and Grafana Cloud is responsible for sampling/aggregating. That may be prohibitively expensive in terms of CPU/network load. What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries. The same thing with metri…

> If I understand that correctly, it means your app always creates traces

Yes, because otherwise what you propose requires modifying the binary in-place and that's too big of a security hole for lots of (production) environments. Some variants of that could work with an out-of-process method like Dtrace or eBPF, but that means mutating the kernel, even more of a no-no.

Re: OTel isn’t going well

#30

Post by veqq »

EdSchouten wrote: post #9

What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime. For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again…

You can do that in Lisp, since you can arbitrarily redefine the wrapper to have such or other logic etc.
Post reply on HN