Live data from Hacker News

OTel isn’t going well

matduggan.com

81–90 of 123 posts

Re: OTel isn’t going well

#81
post #34

Earlier quoted context omitted.

I just don’t get this sentiment. How would you represent metrics as traces? You cannot. Even reconstructing traces from logs would be challenging at best. How would you get, say, Garbage Collector metrics from logs or traces? You cannot. There is no magic bullet. Observability isn’t something you can just slap on and call it a day. While traces and logs might share superficial similarities, they are not the same. And…

> How would you represent metrics as traces? Just instrument your meter implementation so each observation produces a span. Boom, free metric-derived traces.

"free". The observability system would greatly exceed the workload being observed in many cases.

Re: OTel isn’t going well

#82
post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

Totally agree. However I am hopeful. We started the first full instrumented project a few years back. It took us a long time to do the whole work including understanding the SDK, mapping the dimensions and getting everything right. Our last project we did the whole thing with agents and they really took away a lot of the pain from the implementation part. We also use Axiom MCP so when we need some trace or event in the logs the agents look for it and if they don’t find it they’ll add it for the next time. It’s really been a different experience.

Re: OTel isn’t going well

#83
post #58

Earlier quoted context omitted.

Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then rec…

This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction. Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with diff…

You are conflating the challenges of ingesting and querying at large scale with the what the original comment is about, which is emitting them more easily.

Re: OTel isn’t going well

#84
post #64

I find the entire observability space to quite a poor experience, at least in the self-hosted space. Tried both grafana route and signoz and neither seems particularly pleasant

What about the experience did you find lacking?

Re: OTel isn’t going well

#85
post #46
post #43

Earlier quoted context omitted.

A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars). The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink. It's just outbound HTTP or GRPC, a…

In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?

I don't use Lambda anymore, but yes. I submitted traces to AWS XRay in a background goroutine with a small timeout.

Re: OTel isn’t going well

#86

I've found their django instrumentation to be kinda useless for larger apps. The only choices you get is full auto instrumentation, which breaks most non-trivial apps, or zero assistance/documentation. There is no in-between where I can inject the functionality required in a way that is compatible with the application.

Not the OP, but turning on auto-instrumentation for a Golang app running in Kubernetes breaks the app if the app is either:

- Running an old version of Golang (older than 1.18 if memory serves), or

- has libraries that the eBPF probes don't like.

And while I like OTel, I agree with the OP that you are absolutely going deep-sea diving if you're going to do anything beyond the examples provided (which is very easy to do!)

Re: OTel isn’t going well

#87

The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage. At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge! Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It…

The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone . This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither. Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated , either, and in fact the fact that they're not c…

And Vector for logs, which is also freaking great.

Re: OTel isn’t going well

#88
post #14

Earlier quoted context omitted.

The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone . This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither. Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated , either, and in fact the fact that they're not c…

OTEL metrics are a bit awkward, but they work just fine with Prometheus. Jaeger uses the OTLP protocol nowadays. So it _is_ OTEL.

I believe you can still use Zipkin with Jaeger

Re: OTel isn’t going well

#89
post #60
post #46

Earlier quoted context omitted.

In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?

Use the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block. [0] https://github.com/open-telemetry/opentelemetry-lambda

That lambda layer comes with an incredibly heavy performance penalty.

It doesn't block, but it does consume compute/memory resources and takes forever to startup[0][1]. To be fair, Rotel is promising in this regard[2].

[0]: https://github.com/open-telemetry/opentelemetry-lambda/issue...

[1]: https://github.com/aws-observability/aws-otel-lambda/issues/...

[2]: https://github.com/rotel-dev/rotel

Re: OTel isn’t going well

#90
post #7

OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today: 1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time. 2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now. 3. Serverles…

So what's the alternative then? (Genuine question, not hypothetical snark.)

- Paying Datadog $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$, or

- Using and configuring a suite of tools (Jaeger for tracing, Vector or Fluentd for logs, Prometeheus for metrics)

Post reply on HN