Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The problem with OpenTelemetry

cra.mr

Post reply on HN
Print view View original

41–50 of 178 posts

Re: The problem with OpenTelemetry

#41

Post by markl42 »

At the risk of hijacking the comments, I've been trying to use OTel recently to debug performance of a complex webpage with lots of async sibling spans, and finding it very very difficult to identify the critical path / bottlenecks.

There's no causal relationships between sibling spans. I think in theory "span links" solves this, but afaict this is not a widely used feature in SDKs are UI viewers.

(I wrote about this here https://github.com/open-telemetry/opentelemetry-specificatio...)

Re: The problem with OpenTelemetry

#42

Post by hahn-kev »

MapleWalnut wrote: post #27
doctorpangloss wrote: post #16

I don’t know what the Sentry guy is really saying - I mean you can write whatever code you want, go for it man. But I do have to “pip uninstall sentry-sdk” in my Dockerfile because it clashes with something I didn’t author. And anyway, because it is completely open source, the flaws in OpenTelemetry for my particular use case took an hour to surmount, and vitally, I didn’t have to pay the brain damage cost most devel…

The Sentry SDK is open source and easy to contribute to in my experience.

Yeah but who wants to contribute to an SDK for a service that you need to pay for? That would be like if Oracle DB was open to contribution

Re: The problem with OpenTelemetry

#43

Post by diurnalist »

markl42 wrote: post #41

At the risk of hijacking the comments, I've been trying to use OTel recently to debug performance of a complex webpage with lots of async sibling spans, and finding it very very difficult to identify the critical path / bottlenecks. There's no causal relationships between sibling spans. I think in theory "span links" solves this, but afaict this is not a widely used feature in SDKs are UI viewers. (I wrote about this…

I don't believe this is a solved problem, and it's been around since OpenTracing days[0]. I do not think that the Span links, as they are currently defined, would be the best place to do this, but maybe Span links are extended to support this in the future. Right now Span links are mostly used to correlate spans causally _across different traces_ whereas as you point out there are cases where you want correlation _within a trace_.

[0]: https://github.com/opentracing/specification/issues/142

Re: The problem with OpenTelemetry

#44

Post by hahn-kev »

tnolet wrote: post #37

A recent example of OTel confusion. I could for the life of me not get the Python integration send traces to a collector. Same URL, same setup same API key as for Nodejs and Go. Turns out the Python SDK expect a URL encoded header, e.g. “Bearer%20somekey” whereas all other SDKs just accept a string with a whitespace. The whole split between HTTP, protobuf over HTTP and GRPC is also massively confusing.

Sounds like a problem with the Python sdk

Re: The problem with OpenTelemetry

#45

Post by aserafini »

the_mitsuhiko wrote: post #29
Jemaclus wrote: post #24

Earlier quoted context omitted.

I actually take the opposite approach. In my experience, well instrumented metrics and finely tuned logs are more actionable than distributed traces! Interesting how that works out.

I believe on the infrastructure side that might be correct. Within applications that doesn’t match my experience. In many cases the concurrent nature of servers makes it impossible to repro issues and narrow down the problem without tracing or trace aware logs.

With only sampled traces though it’s very hard to understand the impact of the problem. There are some bad traces but is it affecting 5%, 10% or 90% of your customers. Metrics shine there.

Re: The problem with OpenTelemetry

#46

Post by MapleWalnut »

hahn-kev wrote: post #42
MapleWalnut wrote: post #27

Earlier quoted context omitted.

The Sentry SDK is open source and easy to contribute to in my experience.

Yeah but who wants to contribute to an SDK for a service that you need to pay for? That would be like if Oracle DB was open to contribution

Sentry provides a great hosted service. You can self host if you like, but it’s nicer to let them do it

Re: The problem with OpenTelemetry

#48

Post by wvh »

I have surveyed this landscape for a number of years, though I'm not involved enough to have strong opinions. We're running a lot of Prometheus ecosystem and even some OpenTelemetry stacks across customers. OpenTelemetry does seem like one of these projects with an ever expanding scope. It makes it hard to integrate parts you like and keep things both computing-wise and mentally lightweight without having to go all-in.

It's not anymore about hey, we'll include this little library or protocol instead of rolling our own, so we can hope to be compatible with a bunch of other industry-standard software. It's a large stack with an ever evolving spec. You have to develop your applications and infrastructure around it. It's very seductive to roll your own simpler solution.

I appreciate it's not easy to build industry-wide consensus across vendors, platforms and programming languages. But be careful with projects that fail to capture developer mindshare.

Re: The problem with OpenTelemetry

#49

Post by hinkley »

markl42 wrote: post #41

At the risk of hijacking the comments, I've been trying to use OTel recently to debug performance of a complex webpage with lots of async sibling spans, and finding it very very difficult to identify the critical path / bottlenecks. There's no causal relationships between sibling spans. I think in theory "span links" solves this, but afaict this is not a widely used feature in SDKs are UI viewers. (I wrote about this…

I was underwhelmed by the max size for spans before they get rejected. Our app was about an order of magnitude too complex for OTEL to handle.

Reworking our code to support spans made our stack traces harder to read and in the end we turned the whole thing off anyway. Worse than doing nothing.

Re: The problem with OpenTelemetry

#50

Post by tnolet »

hahn-kev wrote: post #42
MapleWalnut wrote: post #27

Earlier quoted context omitted.

The Sentry SDK is open source and easy to contribute to in my experience.

Yeah but who wants to contribute to an SDK for a service that you need to pay for? That would be like if Oracle DB was open to contribution

Sentry is self hostable. https://develop.sentry.dev/self-hosted/
Post reply on HN