Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OTel isn’t going well

matduggan.com

Post reply on HN
Print view View original

71–80 of 123 posts

Re: OTel isn’t going well

#71

Post by kalkin »

osener wrote: post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

The article assumes the issue with OTel is slow feature development, which isn't my experience at all. The issue I've had is that the SDKs have terrible performance overhead for instrumentation and are, as you say, highly resistant to integrating the output of better performing (or just preexisting) instrumentation. In Python and Ruby, at least, the CPU cost of all the mandatory abstraction is way too high.

Re: OTel isn’t going well

#72

Post by rm »

gertburger wrote: post #55
rm wrote: post #38

Earlier quoted context omitted.

Could you please elaborate a bit on what is not working for you?

(I haven't attempted to use opentelemetry-instrumentation-django in at least a year so my information might be dated and my memory is patchy :P) If I recall the primary issue was the forced loading of the django settings file by otel. I get that fully automated instrumentation should be turn-key and the current approach kinda works on basic applications. But most production django applications are monoliths and gener…

Thanks for the write up, appreciated. A couple of things: - users are not forced to use auto-instrumentation. People can import the Middleware and use it as they see fit. I see that the instrumentor is configuring the middleware using some private attributes, I guess that can be extracted into a public function so it would be easier to do so - speaking of the middleware, the chances that it'll become a public symbol are scarce as are the chances that the interfaces will change. So if one has some testing before going to production it should be fine

Re: OTel isn’t going well

#73

Post by huksley »

OTel is very complicated while yeah for example datadog is just dropin. And Graylog support for OTel makes it a second class citizen in the logs (all attributes are prepended with otel_attributes_ which makes searching difficult).

Using is hard, vendors are hostile, it seems like no-one want it to be a first class citizen...

Re: OTel isn’t going well

#75

Post by PunchyHamster »

osener wrote: post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

Even just basic wire protocol is ass that's PITA to parse, like list of attibutes (which have to be unique) isn't a map but array of maps with some weird way to encode key and type. The whole project is industrial scale mediocrity

Re: OTel isn’t going well

#76

Post by PunchyHamster »

twic wrote: post #58
Flamkuchlo wrote: post #40

Earlier quoted context omitted.

The problem is not the instrumentation but the way everyone of them work. A metric is a point in time. A metric is very small but you have a lot of them. A log is when something is happening but you need to log it out. A logline is heavy and has a lot of context. User id, message, etc. A trace needs to start at the request level and tracing until the response. This is the slowest and heaviest operation. How do you de…

Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then rec…

> Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events.

No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like returning app's CPU/memory usage)

Re: OTel isn’t going well

#77

Post by PunchyHamster »

thorian1828i03 wrote: post #10
EdSchouten wrote: post #9

What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime. For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again…

One strategy do to do that is to trace everything by default and select what to sample later, e.g. https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...

It is very easy way to have your tracing infrastructure cost more than actual infrastructure.

Re: OTel isn’t going well

#78

Post by Kinrany »

It feels like OTel tried standardizing before the correct design was anywhere close to being settled. It's only time to standardize once there's consensus on all the important points, and what's left is minor details that don't matter for anything other than compatibility.

Re: OTel isn’t going well

#79

Post by jandrewrogers »

twic wrote: post #58
Flamkuchlo wrote: post #40

Earlier quoted context omitted.

The problem is not the instrumentation but the way everyone of them work. A metric is a point in time. A metric is very small but you have a lot of them. A log is when something is happening but you need to log it out. A logline is heavy and has a lot of context. User id, message, etc. A trace needs to start at the request level and tracing until the response. This is the slowest and heaviest operation. How do you de…

Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then rec…

This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction.

Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with different patterns of traversal over raw events. As data model, you need to build secondary indexes over the raw metrics to reflect the orthogonal data access patterns depending on if you are evaluating them as logs, metrics, or traces. This famously has poor scalability and performance.

In analytical processing we largely manage the inherent performance and scalability issues using denormalization, which allows processing pipelines with very different requirements to be optimized independently. Or in this context, treating logs, metrics, and traces as unrelated things with independent infrastructure.

"Observability 2.0" deeply embeds an architectural assumption that all systems are small. It is not a tractable architecture in high-scale or high-performance systems.

Real silicon has a long history of destroying beautiful conceptual abstractions in software engineering.

Re: OTel isn’t going well

#80

Post by jandrewrogers »

osener wrote: post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

OpenTelemetry reminds me a lot of the bad old days when Java/XML maximalism was fashionable.
Post reply on HN