Earlier quoted context omitted.
I just don’t get this sentiment. How would you represent metrics as traces? You cannot. Even reconstructing traces from logs would be challenging at best. How would you get, say, Garbage Collector metrics from logs or traces? You cannot. There is no magic bullet. Observability isn’t something you can just slap on and call it a day. While traces and logs might share superficial similarities, they are not the same. And…
> How would you represent metrics as traces? Just instrument your meter implementation so each observation produces a span. Boom, free metric-derived traces.
OTel isn’t going well
81–90 of 123 posts
Re: OTel isn’t going well
#82I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…
Re: OTel isn’t going well
#83Earlier quoted context omitted.
Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then rec…
This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction. Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with diff…
Re: OTel isn’t going well
#84I find the entire observability space to quite a poor experience, at least in the self-hosted space. Tried both grafana route and signoz and neither seems particularly pleasant
Re: OTel isn’t going well
#85Earlier quoted context omitted.
A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars). The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink. It's just outbound HTTP or GRPC, a…
In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?
Re: OTel isn’t going well
#86I've found their django instrumentation to be kinda useless for larger apps. The only choices you get is full auto instrumentation, which breaks most non-trivial apps, or zero assistance/documentation. There is no in-between where I can inject the functionality required in a way that is compatible with the application.
- Running an old version of Golang (older than 1.18 if memory serves), or
- has libraries that the eBPF probes don't like.
And while I like OTel, I agree with the OP that you are absolutely going deep-sea diving if you're going to do anything beyond the examples provided (which is very easy to do!)
Re: OTel isn’t going well
#87The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage. At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge! Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It…
The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone . This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither. Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated , either, and in fact the fact that they're not c…
Re: OTel isn’t going well
#88Earlier quoted context omitted.
The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone . This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither. Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated , either, and in fact the fact that they're not c…
OTEL metrics are a bit awkward, but they work just fine with Prometheus. Jaeger uses the OTLP protocol nowadays. So it _is_ OTEL.
Re: OTel isn’t going well
#89Earlier quoted context omitted.
In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?
Use the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block. [0] https://github.com/open-telemetry/opentelemetry-lambda
It doesn't block, but it does consume compute/memory resources and takes forever to startup[0][1]. To be fair, Rotel is promising in this regard[2].
[0]: https://github.com/open-telemetry/opentelemetry-lambda/issue...
[1]: https://github.com/aws-observability/aws-otel-lambda/issues/...
Re: OTel isn’t going well
#90OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today: 1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time. 2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now. 3. Serverles…
So what's the alternative then? (Genuine question, not hypothetical snark.)
- Using and configuring a suite of tools (Jaeger for tracing, Vector or Fluentd for logs, Prometeheus for metrics)