Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OTel isn’t going well

matduggan.com

Post reply on HN
Print view View original

41–50 of 123 posts

Re: OTel isn’t going well

#41

Post by clintonb »

bilalq wrote: post #26
cyberax wrote: post #19

Earlier quoted context omitted.

> 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage. You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent. App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the UR…

Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads. Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to. In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and the…

(I’m not the person you replied to, but have experience here.)

I follow the [gateway deployment pattern](https://opentelemetry.io/docs/collector/deploy/gateway/). Everything sends telemetry to our gateway, which exports to ClickHouse (formerly Datadog).

We use Node.js, so all we need to do is run a script initializing Otel before running the app. We set this up following the docs a few years ago, and haven’t had to change it much since then.

Re: OTel isn’t going well

#42

Post by sweetgiorni »

fuzzy2 wrote: post #34
EdSchouten wrote: post #9

What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime. For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again…

I just don’t get this sentiment. How would you represent metrics as traces? You cannot. Even reconstructing traces from logs would be challenging at best. How would you get, say, Garbage Collector metrics from logs or traces? You cannot. There is no magic bullet. Observability isn’t something you can just slap on and call it a day. While traces and logs might share superficial similarities, they are not the same. And…

> How would you represent metrics as traces?

Just instrument your meter implementation so each observation produces a span. Boom, free metric-derived traces.

Re: OTel isn’t going well

#43

Post by cyberax »

bilalq wrote: post #26
cyberax wrote: post #19

Earlier quoted context omitted.

> 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage. You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent. App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the UR…

Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads. Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to. In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and the…

A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars).

The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink.

It's just outbound HTTP or GRPC, and it doesn't have to go over public Internet.

> In a runtime like Lambda, how would you avoid the need to run an edge collector?

Here's my setup (in Go, very simplified):

> // Instantiate a new slog logger > logger := otelslog.NewLogger("root", otelslog.WithLoggerProvider(otelLogger)) > // Use the logger as needed

My code uses proper Go loggers exclusively. I also redirected the stdout and stderr to a goroutine (via the usual close(2)+open() trick) to serve as a catch-all sink for anything that slips the net.

Re: OTel isn’t going well

#44

Post by nlitened »

tete wrote: post #39

OpenTelemtry is the perfect example of an overengineered mess. While I usually think that at least having some standard that people agree on I think OpenTelemtry should be dropped. A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards and a lot better both in terms of less added complexity and the results you get. I think OpenTelemetry turne…

I agree overall, however:

> A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards

By all metrics (hah), Prometheus is the more popular solution and is the de-facto standard, as far as I know.

Re: OTel isn’t going well

#45

Post by cyberax »

arcanemachiner wrote: post #22
cyberax wrote: post #14

Earlier quoted context omitted.

OTEL metrics are a bit awkward, but they work just fine with Prometheus. Jaeger uses the OTLP protocol nowadays. So it _is_ OTEL.

What, so people don't like OTel, but they like Jaeger, which implements an OTel spec? (I'm a noob to this subject, if that wasn't obvious.)

Yep.

Kinda like people hating Obamacare but loving the ACA.

Jaeger does not implement all the OTEL features, though. It's specifically focused on traces rather than metrics.

Re: OTel isn’t going well

#46

Post by bilalq »

cyberax wrote: post #43
bilalq wrote: post #26

Earlier quoted context omitted.

Perhaps there's a gap in my understanding. Can you clarify on this a bit more? I run a mix of serverless and non-serverless workloads. Gateway collectors are unavoidable because various SaaS platforms require you to be running publicly reachable endpoints to send telemetry to. In a runtime like Lambda, how would you avoid the need to run an edge collector? The only thing that comes to mind is to write to logs and the…

A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars). The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink. It's just outbound HTTP or GRPC, a…

In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?

Re: OTel isn’t going well

#47

Post by fallingbananna »

sweetgiorni wrote: post #42
fuzzy2 wrote: post #34

Earlier quoted context omitted.

I just don’t get this sentiment. How would you represent metrics as traces? You cannot. Even reconstructing traces from logs would be challenging at best. How would you get, say, Garbage Collector metrics from logs or traces? You cannot. There is no magic bullet. Observability isn’t something you can just slap on and call it a day. While traces and logs might share superficial similarities, they are not the same. And…

> How would you represent metrics as traces? Just instrument your meter implementation so each observation produces a span. Boom, free metric-derived traces.

Yup. Not a difficult problem to solve.

In the code define everything as a span with a name, scope (start-end), description and tags... and then you can easily dynamically produce traces, spans, logs or metrics based on what you need.

Re: OTel isn’t going well

#48

Post by hobofan »

EdSchouten wrote: post #9

What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime. For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again…

I don't think OTEL is necessarily "at fault" here. It's a split that's carried all throughout the observability ecosystem. e.g. in the Grafana suite of solutions you have Loki (logs), Tempo (tracing) and Mimir (metrics) to cover storage & querying for all three axis, as all of them have very distinct processing & performance characteristics.

While it may intuitively may look like there is a large overlap in the three areas there is suprisingly little, and for the few parts there are (e.g. trace log correlation), OTEL does offer a standard.

Re: OTel isn’t going well

#49

Post by spockz »

EdSchouten wrote: post #9

What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime. For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again…

I think it is almost a inevitability where otel came as a standardised aggregate of OpenTracing (which was the same but only for tracing over multiple tracing implementations), logging, and metrics into a single observability standard without alienating all the individual supporting vendors.

Historically, logging and metrics have been different problem domains with different implementations for ages.

Now to your point: Note that tracing does get the most of love, and that it does include constructs to add logging and metrics into these traces (spans actually). So you could argue that they are trying to develop a single interface.

> “The next time something like this occurs again, please save me a trace.”

Well, if you want this you either need to propagate this predicate to all points that might be involved, or always emit all traces and have the predicate included in the filter. And then you need to be able to dynamically propagate this predicate from the system/ui where you click to where you filter.

This is one of the reasons why we always propagate and emit traces and just post filter it in processing before it lands in the persistence layer.

Re: OTel isn’t going well

#50

Post by spockz »

Flamkuchlo wrote: post #40
anygivnthursday wrote: post #36

Earlier quoted context omitted.

I am not sure if this is what they mean, but e.g. with Micrometer in Java you can instrument your code once with observations that produces observation events, then you can register handlers that can turn them into metrics, or logs, or traces without having to instrument your code three times. https://docs.micrometer.io/micrometer/reference/observation....

The problem is not the instrumentation but the way everyone of them work. A metric is a point in time. A metric is very small but you have a lot of them. A log is when something is happening but you need to log it out. A logline is heavy and has a lot of context. User id, message, etc. A trace needs to start at the request level and tracing until the response. This is the slowest and heaviest operation. How do you de…

Technically, you can use the same places in the code where you stop/start/fork traces to also be the places where you increment the counters/gauges, etc. Which I think the GP was alluding to when describing the micrometer solution. Similarly, you can derive metrics for log lines without having to emit the actual log lines.

Then separately you can have log levels or verbosity levels that control to which level you actually emit traces/logs and/or roll up metrics.

Post reply on HN