Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OTel isn’t going well

matduggan.com

Post reply on HN
Print view View original

101–110 of 123 posts

Re: OTel isn’t going well

#101

Post by Groxx »

cogman10 wrote: post #98
Groxx wrote: post #94

Earlier quoted context omitted.

Yeah, they spent a ton of effort trying to cram automatic-config-and-library-discovery-like features everywhere when they would've been MUCH better served by requiring explicit dependency injection... and then just adding DI wrappers externally. That's what contrib is for. As it stands, due to the tower of abstractions that could've just been "init with an implementation of this interface", you need to learn several…

It's crazy, IMO, that they didn't simply design otel clients. Making this giant cross language framework is an insane endeavor that just makes everyone unhappy.

Yeah, cross language is generally considered "a protocol", and exists only to cross process boundaries. That's definitely useful! It's even a mostly reasonable one (though with a few weird decisions either due to blindly copying Prometheus' flaws or due to... idk avoiding copying Prometheus on principle? Very strange sometimes, but livable). We needed a grand unification here, even if mediocre, and the time was right.

Trying to make all the supported languages feel similar (beyond sharing concepts which almost directly match the protocol) is foolish in the extreme, and it's why it's such a monstrosity. And worse, they seem to treat that as more important than the bottom-most clients that speak the protocol, so you might be waiting years for any support for a third of the system!

Re: OTel isn’t going well

#102

Post by chrismarlow9 »

cyberax wrote: post #13

I disagree. I'm an observability geek, and OTel is... fine. It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements: 1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" quer…

Agreed. Otel itself is fine. The documentation is bad though and full of inconsistent best practices and examples that are flat out wrong and other things.

My life of working with it got easier when I started just looking at the actual code, using network level tools like nc/tcpdump, making extensive use of the debug exporter, and almost ignoring the docs entirely except as a basic summary of what a thing does.

Re: OTel isn’t going well

#103

Post by jandrewrogers »

skrtskrt wrote: post #83
jandrewrogers wrote: post #79

Earlier quoted context omitted.

This is the literally the "everything is a graph" argument from database architecture. The conceptual abstraction fails badly because it has to be implemented on real silicon that imposes constraints not considered in the abstraction. Logs, metrics, and traces are all derived from raw events but none of them are intrinsically discrete events in a systems engineering sense. They are all different data models with diff…

You are conflating the challenges of ingesting and querying at large scale with the what the original comment is about, which is emitting them more easily.

I don't see them as separate issues. Emitting them directly runs into the inherently poor memory locality (and potentially concurrency) of trying to produce logs, metrics, and traces from the same underlying event data representation.

It is only "easy" if performance and scalability don't matter.

Re: OTel isn’t going well

#104

Post by luckydata »

czhu12 wrote: post #95

It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host. Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.

Signoz

Re: OTel isn’t going well

#105

Post by NegativeLatency »

rcleveng wrote: post #23

Sounds a lot like K8s. It's not a framework you use, it's a framework to build a framework on top of. I wish the observability vendors would move to using it under the covers so it's easier to mix and match. I wish the otel support wasn't super buggy in most of the frameworks and backends.

But then you wouldn’t be locked in!

Re: OTel isn’t going well

#106

Post by krab »

PunchyHamster wrote: post #76
twic wrote: post #58

Earlier quoted context omitted.

Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then rec…

> Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like return…

The app's memory usage is an aggregation of the alloc/free events. I think the original point was that all of the metrics, traces and logs are conceptually the same but for efficiency, we store less data in each place, not the full history. Personally, for the systems I work on, having an easy way to turn logs into metrics and vice versa, without deciding up front, would be a slight benefit.

Re: OTel isn’t going well

#107

Post by dwoldrich »

I think the industry would benefit from some general evangelism for observability. Being able to do distributed tracing was both a "well, duh" and mindblown experience when I first learned about it a decade ago. It made supporting software so much better.

OTel is a fine system for learning observability; it does an okay job of exposing capabilities given how diverse the vendor ecosystem is.

Re: OTel isn’t going well

#108

Post by masterj »

steerpike wrote: post #28
masterj wrote: post #21

Earlier quoted context omitted.

HN always grumbles about OTel, but I agree. It's fine, and important: https://jeremymorrell.dev/blog/opentelemetry-and-the-value-o... I understand the author's perspective in the linked article, but none of that data shows a project in trouble? Some languages have more resources than others, but those all look like healthy open source projects

Oh my god. A Jeremy Morrell sighting in the wild. Every time I share your blog (and I share it a lot ) I tell people: "This guy started a blog in 2024. Wrote three posts and all three of them would still make my top ten list of 'greatest posts on observability' today". 'A practitioner's guide to wide events' especially is still my number 1.

D'aww, thank you! I'm hoping to find time to write more this year

Re: OTel isn’t going well

#109

Post by masterj »

dijit wrote: post #91

I know sadly very little about otel, it feels “heavy” in a way I am not used to, I am used to simple systems - configured and composed in a way that makes a larger system. 20 years ago, we were doing (what I think) OTel is doing: with “hit IDs” (half way between a session and a request) that were consistently applied when logging the cause a request being fired; along centralised logging and really good timekeeping.…

You are essentially describing a proto-tracing system. At the risk of self-promoting twice in one comments section, I have a post walking through going from what you describe above to OTel-compatible tracing: https://jeremymorrell.dev/blog/minimal-js-tracing/

You are right that what you were doing is very similar! However standardization helps a lot here.

Re: OTel isn’t going well

#110

Post by tablloyd »

> However on the collector side you end up having to do the OpenTelemetry Collector Builder to make your own collector (or just kinda ride the wave and hope it works out). While cool that this exists, it's a lot of scope to ask a team to take on.

This is just plain wrong, binaries of the collector are shipped which are available to use straight away. You can use the builder if you want to create your own version with a selected set of components but it is no way a hard requirement.

Post reply on HN