Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OTel isn’t going well

matduggan.com

Post reply on HN
Print view View original

11–20 of 123 posts

Re: OTel isn’t going well

#11

Post by 0xbadcafebee »

It is crazy to me how often people don't grok how to design software well.

1. The worst thing you can do is try to stuff too many things into one specification. So you want an API? That's great. What's that? You want a rigid set of types so that any tiny changes over time aren't compatible? You want to try to define every conceivable use case as a new call? You want to combine multiple elements from different domains into one flat set of functions? You don't have any hierarchy or inheritance? You don't support extensions?

2. The second-worst thing you can do is to force a whole lot of different people to go through a single standards body. So you want to support a thousand different 3rd party components. What's that? You want to require everyone get their adapter approved by one group? And there's only one supported adapter per 3rd party component?

If you're trying to feed an entire city, it's logistically incredibly difficult to try to do it all yourself. If instead you just define where food can be dropped off or picked up, and ask volunteers to bring their own food there whenever they can/want, now you don't have a logistical nightmare on your hands anymore. The tech alternative? Add support for "plugins", make the plugin interface incredibly loose/backwards-compatible/layered, and invite people to publish their own plugins. If you under-engineer it, it actually works better.

Re: OTel isn’t going well

#12

Post by EdSchouten »

thorian1828i03 wrote: post #10
EdSchouten wrote: post #9

What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime. For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again…

One strategy do to do that is to trace everything by default and select what to sample later, e.g. https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...

If I understand that correctly, it means your app always creates traces, and Grafana Cloud is responsible for sampling/aggregating. That may be prohibitively expensive in terms of CPU/network load.

What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries.

The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).

Re: OTel isn’t going well

#13

Post by cyberax »

I disagree. I'm an observability geek, and OTel is... fine.

It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements:

1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" queries, etc.

2. Ability to downgrade errors for segments that are expected to return an error (e.g. HEAD on a non-existing object in S3 to check if there's a cached blob).

Re: OTel isn’t going well

#14

Post by cyberax »

GauntletWizard wrote: post #5
jiggawatts wrote: post #4

The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage. At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge! Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It…

The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone . This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither. Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated , either, and in fact the fact that they're not c…

OTEL metrics are a bit awkward, but they work just fine with Prometheus.

Jaeger uses the OTLP protocol nowadays. So it _is_ OTEL.

Re: OTel isn’t going well

#15

Post by thorian1828i03 »

EdSchouten wrote: post #12
thorian1828i03 wrote: post #10

Earlier quoted context omitted.

One strategy do to do that is to trace everything by default and select what to sample later, e.g. https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...

If I understand that correctly, it means your app always creates traces, and Grafana Cloud is responsible for sampling/aggregating. That may be prohibitively expensive in terms of CPU/network load. What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries. The same thing with metri…

> The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add).

How does the monitoring system have any of the context to add labels? That would only exist in application memory.

Grafana went the other way - your app exports all labels, and then you selectively aggregate on ingest: https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...

> That may be prohibitively expensive in terms of CPU/network load.

In practice I've not experienced this even on quite high request rates. While it isn't free, exporting everything has been cheap enough that the real cost in dollars spent is basically marginal (it's _storing_ the data that's expensive)

Re: OTel isn’t going well

#16

Post by morganherlocker »

GauntletWizard wrote: post #5
jiggawatts wrote: post #4

The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage. At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge! Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It…

The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone . This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither. Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated , either, and in fact the fact that they're not c…

Prometheus is so easy to add and if you need more scale, there is mimir and a few other options with similar client semantics. I really can't imagine reaching for a framework APK that tries to anticipate every possible thing I would want telemtered, and is inevitably missing all the domain specific derived channels I need. Even prepackaged Prometheus exporters are usually overkill.

Re: OTel isn’t going well

#17

Post by EdSchouten »

thorian1828i03 wrote: post #15
EdSchouten wrote: post #12

Earlier quoted context omitted.

If I understand that correctly, it means your app always creates traces, and Grafana Cloud is responsible for sampling/aggregating. That may be prohibitively expensive in terms of CPU/network load. What I’m suggesting is that your apps by default only send metrics to your monitoring system, but that the monitoring system can specifically ask to “upgrade” metrics to traces. Or to log entries. The same thing with metri…

> The same thing with metric cardinality: by default, only report metrics in a fully aggregated manner. But do tell the monitoring system how they can potentially be broken up if needed (i.e., which labels to add). How does the monitoring system have any of the context to add labels? That would only exist in application memory. Grafana went the other way - your app exports all labels, and then you selectively aggrega…

> How does the monitoring system have any of the context to add labels? That would only exist in application memory.

Indeed. If you have a protocol that doesn’t allow exposing that kind of information, then that only lives in application memory. But my suggestion is that it’s exposed.

Re: OTel isn’t going well

#18

Post by lenkite »

GauntletWizard wrote: post #5
jiggawatts wrote: post #4

The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage. At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge! Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It…

The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone . This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither. Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated , either, and in fact the fact that they're not c…

Hard Agree on Prometheus. And esp on the complexity - OTel is dizzyingly complex. You can get started ASAP on Prometheus whereas you get lost in analysis-paralysis when dealing with OTel.

Re: OTel isn’t going well

#19

Post by cyberax »

bilalq wrote: post #7

OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today: 1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time. 2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now. 3. Serverles…

> 4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.

You most certainly don't. You can run your app (especially if it's "serverless") without the collector agent.

App-to-agent and agent-to-sink use the same protocol, so all you need to do is set up the tracing/logging/metrics exporters to directly speak with the sink. These days, it typically means specifying the URL and the DSN header.

Re: OTel isn’t going well

#20

Post by arcanemachiner »

bilalq wrote: post #7

OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today: 1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time. 2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now. 3. Serverles…

So what's the alternative then? (Genuine question, not hypothetical snark.)
Post reply on HN