Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The problem with OpenTelemetry

cra.mr

Post reply on HN
Print view View original

121–130 of 178 posts

Re: The problem with OpenTelemetry

#121

Post by codereflection »

zeeg wrote: post #78
codereflection wrote: post #74

I understand what the author is saying, but vendor lock-in with closed-source observability platforms is a significant challenge, especially for large organizations. When you instrument hundreds or thousands of applications with a specific tool, like the Datadog Agent, disentangling from that tool becomes nearly impossible without a massive investment of engineering time. In the Platform Engineering professional serv…

I totally agree I just wish we could do it in a way that doesn’t try to lump every problem into the same bucket. I don’t see what it achieves personally, and I think it’s limiting the ability for the original goals of the project to be as successful as they could be.

Well, telemetry is defined as logs, metrics, traces... So it kinda makes sense that OTEL supports the major aspects of telemetry.

Re: The problem with OpenTelemetry

#123

Post by ankitnayan »

I think all of us agree that OpenTelemetry's end-goal of making Observability vendor neutral is futuristic and inevitable. We can complain about it being hard to get started, bloated, etc but the value it provides is clear, esp, when you are paying $$$ to a vendor and stuck with it.

OpenStandards also open up a lot of usecases and startups too. SigNoz, TraceTest, TraceLoop, Signadot, all are very interesting projects which OpenTelemetry enabled.

The majority of the problem seems like sentry is not able to provide it's sentry like features by adopting otel. Getting involved at the design phase could have helped shaped the project that could have considered your usecases. The maintainers have never been opposed to such contributions AFAIK.

Regarding, limiting otel just to tracing would not be sufficient today as the teams want a single platform for all observability rather than different tools for different signals.

I have seen hundreds of companies switch to opentelemetry and save costs by being able to choose the best vendor supporting their usecases.

lack of docs, learning curve, etc are just temporary things that can happen with any big project and should be fixed. Also, otel maintainers and teams have always been seeking help in improving docs, showcasing usecases, etc. If everyone cares enough for the bigger picture, the community and existing vendors should get more involved in improving things rather than just complaining.

Re: The problem with OpenTelemetry

#124

Post by lloeki »

yunwal wrote: post #114
chipdart wrote: post #68

Earlier quoted context omitted.

> This is my main gripe too. I don't understand why {traces, logs, metrics} are not just different abstractions built on top of "events" (blobs of data your application ships off to some set of central locations). By design, they cannot be abstractions of the single concept. For example, logs have a hard requirement on preserving sequential order and session and emitting strings, whereas metrics are aggregated and sa…

> Just because you can squint hard enough to only see events being emitted If you squint hard enough you can fool yourself into thinking all metrics have the same availability requirements. It’s not the case. There are plenty of time series data metrics where arbitrarily dropping them or aggregating them would throw off your alerting entirely.

Indeed one would have to squint to the point of blindness.

Logs are single point in time, flat, linear sequence, never dropped (at best you'd collapse sequences of identical, repeated logs). Think dmesg, syslog, systemd journald/journalctl.

Metrics are statistical numeric data, which can be series, average, histogram, bucket... aggregation/reduction can be done on the fly/before leaving the observed thing. Some can be dropped, but it is important that dropping anything stays statistically meaningful.

Spans are a duration in time representing some operation, with metadata (numeric, stringy, structured even) attached pertaining to that operation. Spans have a parent, forming a tree, which forms a trace. Spans can be deduped and/or sampled, with specific occurences forcefully kept (e.g 500 error) or dropped (e.g healthcheck).

They are fundamentally different (technical) primitives a.k.a (functional) tools to observe different things and serve different goals.

Re: The problem with OpenTelemetry

#126

Post by zeeg »

merb wrote: post #116
zeeg wrote: post #69

Earlier quoted context omitted.

Author here. Y’all realize we’d just make more money if everyone has better instrumentation and we could spend less time on it, and more time on the product, right? There is no conspiracy. It’s simple math and reasoning. We don’t compete with most otel consumers. I don’t know how you could read what I posted and think sentry believes otel is a threat, let alone from the fact that we just migrated our JS SDK to run of…

I use sentry selfhosted and started with Otel under gcloud and grafana and everything worked. Perfectly even, with low effort in dotnet. However with sentry it’s still a pain and the visualization in sentry is kinda weird, since it goes beyond tracing. And since sentry itself has no otel endpoint it is also really hard to do things like tail sampling.

Can you give more context on what you mean around sampling. What does an OTLP endpoint have to do with it?

Re: The problem with OpenTelemetry

#127

Post by chipdart »

miohtama wrote: post #72
nimih wrote: post #71

Earlier quoted context omitted.

Sentry is technically self-hostable, but they provide no deployment guidance beyond running the giant blob of services/microservices (including instances of postgres, redis, memcache, clickhouse, and kafka) as a single docker-compose thing. I get why they do this and think it's totally reasonable of them, but Sentry is a very complicated piece of software and takes substantially more work IME to both get up and runni…

Our Linux devops engineer, who had not used Sentry before, set up a self-hosted Sentry in a day.

> Our Linux devops engineer, who had not used Sentry before, set up a self-hosted Sentry in a day.

I've also spent 1 hour setting up a Kubernetes cluster on a set of desktops I had lying around. This does not mean Kubernetes is simple or easy.

Re: The problem with OpenTelemetry

#128

Post by zeeg »

codereflection wrote: post #121
zeeg wrote: post #78

Earlier quoted context omitted.

I totally agree I just wish we could do it in a way that doesn’t try to lump every problem into the same bucket. I don’t see what it achieves personally, and I think it’s limiting the ability for the original goals of the project to be as successful as they could be.

Well, telemetry is defined as logs, metrics, traces... So it kinda makes sense that OTEL supports the major aspects of telemetry.

Says who? Sentry has many other types of telemetry and we’ve existed long before OTel. Who are these all knowing humans who say this is what telemetry is? Are they also going to build every collector for every kind of past current telemetry?

The whole idea that some marketing bs has translated to technology fact is why we’re in this mess.

Re: The problem with OpenTelemetry

#129

Post by mikeshi42 »

wdb wrote: post #22

Personally, I like OpenTelemetry, nice standardised approach. I just wished the vendors would have better support for the semantic conventions defined for a wide variety of traces. I quite like the idea of only need to change one small piece of the code to switch otel exporters instead of swapping out a vendor trace sdk. My main gripe with OpenTelemetry I don't fully understand what the exact difference is between (t…

It's a bit confusing but here's my best attempt to explain it:

- Trace events (span events) are intended to be structured events and possibly can have semantic attributes behind them - similar to how spans have semantic attributes. They're great if your team is all bought in on tracing as an organization. They will colocate your span events with your parent span. In practice they have poor searchability/indexing in many tools, so they should only be used if you only intend to use them when you will discover the span first. (Ex. debug info that is only useful to figure out why a span was very slow and you're okay not being easily searchable)

- Log records are plain old logs, they should be structured, but don't have to be, and there isn't a high expectation of structured data, much less semantic attributes. Logs can be easily adopted without buying into tracing.

- Events API, this is an experimental part of Otel, but is intended to be an API that emits logs with the expectation of semantic conventions (and therefore is also structured). Afaik end users are not the intended audience of this API.

Many teams fall along the spectrum of logs vs tracing which is why there's options to do things multiple ways. My personal take is that log records are going to continue to be more flexible than span events as an end-user given the state of current tools.

Disclaimer: I help build hyperdx, we're oss, otel-based observability and we've made product decisions based on the above opinions.

Re: The problem with OpenTelemetry

#130

Post by mikeshi42 »

cogman10 wrote: post #14

Perhaps the real problem with OTel (IMO) is it's trying to be everything for everyone and every language. It's trying to have a common interface so that you can write OTel in Java or Javascript, python or rust, and you basically have the exact same API. I suspect OP is seeing this directly when talking about the cludgyness of the Javascript API.

The Otel spec does give leeway for language-specific details, and the SDKs are not as uniform as you'd expect (ex. Java's agent configuration is very different from Node's auto instrumentation). I'm not denying that there's SDK specs to adhere to, but the abstraction complexity in Otel is really from the amount of flexibility they've tried to build into the SDK for better or for worse.

The flexibility benefits vendors (I work for HyperDX, based on otel) - as it allows for a lot of points of extensibility to build a better experience for end users by extending the vanilla SDK functionality. However, it creates a lot of overhead for end-users trying to adopt the "vanilla" SDKs out of the box as there's 5 layers of abstractions that need to be understood before getting things started (which is bad!)

I've only seen the DX of Otel improve over time across the ecosystems they support - so I suspect we'll get there soon enough.

Post reply on HN