OTel isn’t going well • Hacker News Board

Re: OTel isn’t going well

#91

Post by dijit » Sat, Aug 22, 2026, 4:37 PM UTC

I know sadly very little about otel, it feels “heavy” in a way I am not used to, I am used to simple systems - configured and composed in a way that makes a larger system.

20 years ago, we were doing (what I think) OTel is doing: with “hit IDs” (half way between a session and a request) that were consistently applied when logging the cause a request being fired; along centralised logging and really good timekeeping. Essentially a unique identifier as a tag that followed the request as it passed through the system.

This was enough to debug basically any problem.

We could even measure the distance between requests of the same “hit” and the total wall-time before it managed to return through the load balancer, so we could track our p99 easily.

Though truthfully we didn't make pretty graphs.

I sometimes wonder what OTel gives me more than this, but I work in games now and lots of these things that work well in webdev do not apply at all to our problems.

Re: OTel isn’t going well

#92

Post by ninkendo » Sat, Aug 22, 2026, 4:57 PM UTC

Kinrany wrote: post #78

It feels like OTel tried standardizing before the correct design was anywhere close to being settled. It's only time to standardize once there's consensus on all the important points, and what's left is minor details that don't matter for anything other than compatibility.

Speaking only from my experience using their rust crates, they have undergone more “code feng shui” than any of our other dependencies. They’re still 0.x and every point release seems to re-imagine things enough to break everything and require substantial rewriting. They don’t even bother describing the motivation for changes, just, you can’t use this type any more, it’s private now. You can’t configure metadata here any more, you have to do it there now. It’s been the most painful dependency of ours by far.

Re: OTel isn’t going well

#93

Post by bilalq » Sat, Aug 22, 2026, 5:01 PM UTC

cyberax wrote: post #85

bilalq wrote: post #46

Earlier quoted context omitted.

In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?

I don't use Lambda anymore, but yes. I submitted traces to AWS XRay in a background goroutine with a small timeout.

If you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you now.

The X-Ray daemon and SDKs are all deprecated now in favor of OTel. Things like enchrichment of resource level traces for things like the DynamoDB client in v3 of the AWS JS SDK don't work with the X-Ray SDK. And they never will now. You're now recommended to use the AWS Distro for OpenTelemetry setup and OTel SDKs. The performance overhead of this is heavy, with big cold-start penalties.

Compare this with how the Datadog layer does adaptive flushing and performs relatively much better. Rotel is also promising in this space. But right now, OTel feels immature and things are being deprecated without the replacement being fully baked.

Re: OTel isn’t going well

#94

Post by Groxx » Sat, Aug 22, 2026, 5:06 PM UTC

osener wrote: post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

Yeah, they spent a ton of effort trying to cram automatic-config-and-library-discovery-like features everywhere when they would've been MUCH better served by requiring explicit dependency injection... and then just adding DI wrappers externally. That's what contrib is for.

As it stands, due to the tower of abstractions that could've just been "init with an implementation of this interface", you need to learn several pieces and how they work together (hint: convoluted and horrifically inefficiently) to modify any piece, and inevitably you learn that to get what you want, you need to swap out a major portion of it... but doing that while maintaining the auto-registry nonsense is a gigantic effort. If it's even possible.

It is the new poster-child for "design by committee". It's horrific. Unfortunately it's also usually the best option in large setups. I greatly approve of the high level goal, but omfg

Re: OTel isn’t going well

#95

Post by czhu12 » Sat, Aug 22, 2026, 5:15 PM UTC

It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host.

Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.

Re: OTel isn’t going well

#96

Post by reactordev » Sat, Aug 22, 2026, 5:18 PM UTC

czhu12 wrote: post #95

It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host. Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.

Signoz?

But yes it seemed like OTel was more interested in being a spec than a tool.

Re: OTel isn’t going well

#97

Post by frez1 » Sat, Aug 22, 2026, 5:23 PM UTC

czhu12 wrote: post #95

It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host. Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.

isn't this exactly what the LGTM stack is?

Re: OTel isn’t going well

#98

Post by cogman10 » Sat, Aug 22, 2026, 5:28 PM UTC

Groxx wrote: post #94

osener wrote: post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

Yeah, they spent a ton of effort trying to cram automatic-config-and-library-discovery-like features everywhere when they would've been MUCH better served by requiring explicit dependency injection... and then just adding DI wrappers externally. That's what contrib is for. As it stands, due to the tower of abstractions that could've just been "init with an implementation of this interface", you need to learn several…

It's crazy, IMO, that they didn't simply design otel clients. Making this giant cross language framework is an insane endeavor that just makes everyone unhappy.

Re: OTel isn’t going well

#99

Post by losingthefight » Sat, Aug 22, 2026, 5:29 PM UTC

czhu12 wrote: post #95

It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host. Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.

I run OSS Grafana with Loki, Prometheus, and Tempo. I use an Alloy sidecar taking in OTEL and scraping logs.feom my Go services and selfhost the stack. Once you need to scale it gets a bit more complicated but it's all still OSS.

The biggest challenge I have is that each data source needs it's own query language, which DD and the like don't. That's why at my day job they went with DD despite the costs. Still OTEL but the querying is the same. We are also looking at Dash0 but for all of my personal and consulting jobs, OSS LGTM/P works good for me.

Re: OTel isn’t going well

#100

Post by pphysch » Sat, Aug 22, 2026, 5:33 PM UTC

czhu12 wrote: post #95

It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host. Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.

There is, it's called VictoriaMetrics/Logs/Traces.

https://victoriametrics.com/