Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The problem with OpenTelemetry

cra.mr

Post reply on HN
Print view View original

21–30 of 178 posts

Re: The problem with OpenTelemetry

#21

Post by the_mitsuhiko »

antonyt wrote: post #6

OTel is flawed for sure, but I don't understand the stance against metrics and logs. Traces are inherently sampled unless you're lighting all your money on fire, or operating at so small a scale that these decisions have no real impact. There are kinds of metrics and logs which you always want to emit because they're mission-critical in some way. Is this a Sentry-specific thing? Does it just collapse these three kind…

> Traces are inherently sampled unless you're lighting all your money on fire

You can burn a lot of money with logs and metrics too. The question is how much value you get for the money you throw on the burning pile of monitoring. My personal belief is that well instrumented distributed tracing is more actionable than logs and metrics. Even if sampled.

(Disclaimer: I work at sentry)

Re: The problem with OpenTelemetry

#22

Post by wdb »

Personally, I like OpenTelemetry, nice standardised approach. I just wished the vendors would have better support for the semantic conventions defined for a wide variety of traces.

I quite like the idea of only need to change one small piece of the code to switch otel exporters instead of swapping out a vendor trace sdk.

My main gripe with OpenTelemetry I don't fully understand what the exact difference is between (trace) events and log records.

Re: The problem with OpenTelemetry

#23

Post by wdb »

shaqbert wrote: post #19

Otel is indeed quite complex. And the docs are not meant for quick wins... Otelbin [0] has helped me quite a bit in configuring and making sense of it, and getting stuff done. [0]: https://www.otelbin.io/

That looks pretty cool! OpenTelemetry Collector configuration files are pretty confusing. Do like the collector, though. Makes it easy to sent a subset of your telemetry to trusted partners.

Re: The problem with OpenTelemetry

#24

Post by Jemaclus »

the_mitsuhiko wrote: post #21
antonyt wrote: post #6

OTel is flawed for sure, but I don't understand the stance against metrics and logs. Traces are inherently sampled unless you're lighting all your money on fire, or operating at so small a scale that these decisions have no real impact. There are kinds of metrics and logs which you always want to emit because they're mission-critical in some way. Is this a Sentry-specific thing? Does it just collapse these three kind…

> Traces are inherently sampled unless you're lighting all your money on fire You can burn a lot of money with logs and metrics too. The question is how much value you get for the money you throw on the burning pile of monitoring. My personal belief is that well instrumented distributed tracing is more actionable than logs and metrics. Even if sampled. (Disclaimer: I work at sentry)

I actually take the opposite approach. In my experience, well instrumented metrics and finely tuned logs are more actionable than distributed traces! Interesting how that works out.

Re: The problem with OpenTelemetry

#25

Post by wdb »

aleph_minus_one wrote: post #8
antonyt wrote: post #6

OTel is flawed for sure, but I don't understand the stance against metrics and logs. Traces are inherently sampled unless you're lighting all your money on fire, or operating at so small a scale that these decisions have no real impact. There are kinds of metrics and logs which you always want to emit because they're mission-critical in some way. Is this a Sentry-specific thing? Does it just collapse these three kind…

> OTel is flawed for sure, but I don't understand the stance against metrics and logs. Even if you don't want to consider the privacy concerns: telemetry wastes quite some data of your internet connection .

I think you are more talking about RUM which isn't yet supported by OpenTelemetry. I think they are working on it.

I am not sure if it will support session replays like some vendors like Sentry or New Relic offer. Technically, I think session replays (rrweb etc) is pretty cool but as a web visitor I am not a fan.

Re: The problem with OpenTelemetry

#26

Post by klabb3 »

drewbug01 wrote: post #17

As a contributor to (and consumer of) OpenTelemetry, I think critique and feedback is most welcome - and sorely needed. But this ain’t it. In the opening paragraphs the author dismisses the hardest parts of the problem (presumably because they are human problems, which engineers tend to ignore), and betrays a complete lack of interest in understanding why things ended up this way. It also seems they’ve completely mis…

No dog in the fight here, but… you're saying that one of the top guys at a major observability shop didn’t understand Open Telemetry, then that’s saying much more about OT than it does about his skills or efforts to understand. After all, his main point is that it’s complex and overengineered, which is the key takeaway for curious bystanders like me, whether every detail is technically correct or not.

> it just reads like someone who […] didn’t understand how something worked - and so instead of figuring it out, they’ve decided that it’s just hot garbage.

And what about average developers asked to “add telemetry” to their apps and libraries? Their patience will be much lower than that.

Not necessarily defending the content (frankly it should have had more examples), but I relate to the sentiment. As a developer, I need framework providers to make sane design decisions with minimal api surface, otherwise I’d rather build something bespoke or just not care.

Re: The problem with OpenTelemetry

#27

Post by MapleWalnut »

doctorpangloss wrote: post #16

I don’t know what the Sentry guy is really saying - I mean you can write whatever code you want, go for it man. But I do have to “pip uninstall sentry-sdk” in my Dockerfile because it clashes with something I didn’t author. And anyway, because it is completely open source, the flaws in OpenTelemetry for my particular use case took an hour to surmount, and vitally, I didn’t have to pay the brain damage cost most devel…

The Sentry SDK is open source and easy to contribute to in my experience.

Re: The problem with OpenTelemetry

#28

Post by zeeg »

drewbug01 wrote: post #17

As a contributor to (and consumer of) OpenTelemetry, I think critique and feedback is most welcome - and sorely needed. But this ain’t it. In the opening paragraphs the author dismisses the hardest parts of the problem (presumably because they are human problems, which engineers tend to ignore), and betrays a complete lack of interest in understanding why things ended up this way. It also seems they’ve completely mis…

Author here.

That’s kind of making my point for me fwiw. It’s too complicated. I consider myself a product person so this is my version of that lens on the problem.

I’m not dismissing the people problem at all - I actually am trying to suggest the technology problem is the easier part (eg a basic spec). Getting it implemented, making it easy to understand, etc is where I see it struggling right now.

Aside this is not just my feedback, it’s a synthesis of what I’m hearing (but also what I believe).

Re: The problem with OpenTelemetry

#29

Post by the_mitsuhiko »

Jemaclus wrote: post #24
the_mitsuhiko wrote: post #21

Earlier quoted context omitted.

> Traces are inherently sampled unless you're lighting all your money on fire You can burn a lot of money with logs and metrics too. The question is how much value you get for the money you throw on the burning pile of monitoring. My personal belief is that well instrumented distributed tracing is more actionable than logs and metrics. Even if sampled. (Disclaimer: I work at sentry)

I actually take the opposite approach. In my experience, well instrumented metrics and finely tuned logs are more actionable than distributed traces! Interesting how that works out.

I believe on the infrastructure side that might be correct. Within applications that doesn’t match my experience. In many cases the concurrent nature of servers makes it impossible to repro issues and narrow down the problem without tracing or trace aware logs.

Re: The problem with OpenTelemetry

#30

Post by cdelsolar »

cogman10 wrote: post #11
aleph_minus_one wrote: post #8

Earlier quoted context omitted.

> OTel is flawed for sure, but I don't understand the stance against metrics and logs. Even if you don't want to consider the privacy concerns: telemetry wastes quite some data of your internet connection .

Hey, this isn't the sort of telemetry we are talking about with OTel. About the only "privacy concern" with otel is that you are probably shipping traces/metrics to a cloud provider for your internal applications. This isn't the sort of telemetry getting baked into ms or google that is used to try and identify personal aspects of individuals, this is data that tells you "Foo app is taking 300ms serving /bar which is…

After I added OTel to an open source project I run, I spent a bit of time arguing with someone about telemetry - they kept saying they didn't opt in and that we need to inform our users about it, etc., and I kept saying no, that's not the same type of telemetry. I wonder how common this misconception is.
Post reply on HN