Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OTel isn’t going well

matduggan.com

Post reply on HN
Print view View original

51–60 of 123 posts

Re: OTel isn’t going well

#51

Post by TylerE »

Flamkuchlo wrote: post #40
anygivnthursday wrote: post #36

Earlier quoted context omitted.

I am not sure if this is what they mean, but e.g. with Micrometer in Java you can instrument your code once with observations that produces observation events, then you can register handlers that can turn them into metrics, or logs, or traces without having to instrument your code three times. https://docs.micrometer.io/micrometer/reference/observation....

The problem is not the instrumentation but the way everyone of them work. A metric is a point in time. A metric is very small but you have a lot of them. A log is when something is happening but you need to log it out. A logline is heavy and has a lot of context. User id, message, etc. A trace needs to start at the request level and tracing until the response. This is the slowest and heaviest operation. How do you de…

At that point you almost might as well just log everything. The decision logic is likely about as complex as just doing it. Then I suppose you have a watchdog task that fires off every, say, 15 minutes or an hour or something, looks at the collected data, and either decides to keep it or trash it while recording a tiny "nothing interesting" datapoint.

Re: OTel isn’t going well

#52

Post by TylerE »

fallingbananna wrote: post #47
sweetgiorni wrote: post #42

Earlier quoted context omitted.

> How would you represent metrics as traces? Just instrument your meter implementation so each observation produces a span. Boom, free metric-derived traces.

Yup. Not a difficult problem to solve. In the code define everything as a span with a name, scope (start-end), description and tags... and then you can easily dynamically produce traces, spans, logs or metrics based on what you need.

At some point your monitoring is burning 10x as much CPU as the actual task...

Re: OTel isn’t going well

#53

Post by time4tea »

Its a shame that the various implementations are pretty horrible. Global state, static methods etc etc.

If you get rid of that, and just pass dependencies around, create some appropriate local abstraction around them.. the tooling, be it datadog or honeycomb does a great job making it useful. Can't really say the same for grafana, but ymmv - depending on budget

Re: OTel isn’t going well

#54

Post by phrotoma »

osener wrote: post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

I tried to emit metrics from a python app using otel once. Gave up and switched to prometheus. What a nightmare.

Re: OTel isn’t going well

#55

Post by gertburger »

rm wrote: post #38
gertburger wrote: post #35

I've found their django instrumentation to be kinda useless for larger apps. The only choices you get is full auto instrumentation, which breaks most non-trivial apps, or zero assistance/documentation. There is no in-between where I can inject the functionality required in a way that is compatible with the application.

Could you please elaborate a bit on what is not working for you?

(I haven't attempted to use opentelemetry-instrumentation-django in at least a year so my information might be dated and my memory is patchy :P)

If I recall the primary issue was the forced loading of the django settings file by otel.

I get that fully automated instrumentation should be turn-key and the current approach kinda works on basic applications.

But most production django applications are monoliths and generally larger apps. They have non-trivial configuration processes which are often multi step and source settings from multiple places.

Otel should not assume it can just randomly load a the django settings at an arbitrary time point in the startup process.

In one of our apps the MIDDLEWARE setting specifically is dynamically generated and re-ordered based on enabled features. That application's startup process also has multiple stages and the initialisation of django occurs much later, after dependant config loaders etc have been initialised.

What would allow us to integrate with opentelemetry-instrumentation-django much more easily is a set of smaller primitives that we can configure and call at the appropriate time.

opentelemetry-instrumentation-django has (had?) a lot of logic hidden inside a large "inject" function which could not easily be extracted into the constituent parts and applied in a compatible manner.

https://github.com/open-telemetry/opentelemetry-python-contr...

Re: OTel isn’t going well

#56

Post by Game_Ender »

bilalq wrote: post #46
cyberax wrote: post #43

Earlier quoted context omitted.

A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars). The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink. It's just outbound HTTP or GRPC, a…

In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?

This is what I have done with CLI apps the directly send to the OTEL vendor. It works great.

Re: OTel isn’t going well

#58

Post by twic »

Flamkuchlo wrote: post #40
anygivnthursday wrote: post #36

Earlier quoted context omitted.

I am not sure if this is what they mean, but e.g. with Micrometer in Java you can instrument your code once with observations that produces observation events, then you can register handlers that can turn them into metrics, or logs, or traces without having to instrument your code three times. https://docs.micrometer.io/micrometer/reference/observation....

The problem is not the instrumentation but the way everyone of them work. A metric is a point in time. A metric is very small but you have a lot of them. A log is when something is happening but you need to log it out. A logline is heavy and has a lot of context. User id, message, etc. A trace needs to start at the request level and tracing until the response. This is the slowest and heaviest operation. How do you de…

Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.

A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then reconstruct a trace from those log entries. If you want to associate multiple spans, and separate log entries, within a trace, you use a shared ID, which is just the same as a context entry for logging.

All three of these pillars are just ways of looking at events. They are not fundamentally different at all. This is a mistaken idea in "Observability 1.0" whose correction is the basis of "Observability 2.0".

The pillars still have their uses, but the choice between them is really a non-functional one - storing a log entry for every event might be too expensive, so just store metrics instead, and index every log entry so it can be correlated with nearby ones might be too expensive, so just store specific traces instead.

Re: OTel isn’t going well

#59

Post by jcmfernandes »

osener wrote: post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

Ran into the same issue and didn't find any willingness in the OTEL gods to close this gap.

Re: OTel isn’t going well

#60

Post by ojkelly »

bilalq wrote: post #46
cyberax wrote: post #43

Earlier quoted context omitted.

A typical setup is to run a separate OpenTelemetry collector process on the same host as the app. The app connects to it via localhost on a standard port (although you can override it using env vars). The collector process then sends the metrics/traces/logs to the observability sink. But there's nothing at all preventing you from sending telemetry directly to the observability sink. It's just outbound HTTP or GRPC, a…

In a lambda runtime, are you blocking client responses until logs/traces/metrics flush?

Use the lambda layer [0] it sends the telemetry after the response is sent, so it doesn’t block.

[0] https://github.com/open-telemetry/opentelemetry-lambda

Post reply on HN