OTel isn’t going well • Hacker News Board

Re: OTel isn’t going well

#121

Post by thiht » Sun, Aug 23, 2026, 8:27 AM UTC

osener wrote: post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

The official Go SDK is a nightmare to use and understand. Just the "getting started"[1] requires adding 10 imports.

[1]: https://opentelemetry.io/docs/languages/go/getting-started/#...

Re: OTel isn’t going well

#122

Post by bilalq » Sun, Aug 23, 2026, 10:32 AM UTC

cyberax wrote: post #118

bilalq wrote: post #93

Earlier quoted context omitted.

If you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you n…

Why do you even _need_ these "layers"? It's a simple RPC protocol that submits data tagged with Span and Trace IDs. That's really all there is to it. You can just submit it directly, without involving any layers.

Because blocking on OTel data to flush before sending a response back is often unacceptable. The layers run a standalone process using the Lambda extension API so they can keep running after your function has responded to a request.

Re: OTel isn’t going well

#123

Post by gcbirzan » Sun, Aug 23, 2026, 6:32 PM UTC

cyberax wrote: post #13

I disagree. I'm an observability geek, and OTel is... fine. It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements: 1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" quer…

We did the same for 1, we have a service that picks up messages from a queue and drops 95% of them. In Python, at least, it was really really ugly. We had the alternative of using refinery, since we're on honeycomb, but didn't want to get stuck with something that only works with one vendor.

I don't understand 2, though, that should be easy? You have access to the span and you can set its status code to 0. With the Python SDK that was trivial for us to do.

The one thing I really hate about the Python SDK is that there's no simple way to say "attach this attribute to the top level span in this app". That way, when you get the user id further down in your app, you can just annotate the top level span with it. Thankfully, honeycomb has any, any2, root, parent (that allows you to filter on any span in the trace, or on the root or parent) but that's slow and again, vendor specific.