Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The problem with OpenTelemetry

cra.mr

Post reply on HN
Print view View original

51–60 of 178 posts

Re: The problem with OpenTelemetry

#51

Post by tnolet »

hahn-kev wrote: post #44
tnolet wrote: post #37

A recent example of OTel confusion. I could for the life of me not get the Python integration send traces to a collector. Same URL, same setup same API key as for Nodejs and Go. Turns out the Python SDK expect a URL encoded header, e.g. “Bearer%20somekey” whereas all other SDKs just accept a string with a whitespace. The whole split between HTTP, protobuf over HTTP and GRPC is also massively confusing.

Sounds like a problem with the Python sdk

Well actually. They (python SDK maintainers) argue their implementation is the correct one according to the spec. See this issue thread for example.

https://github.com/open-telemetry/opentelemetry-specificatio...

There are more. This is a symptom of a how hard it is to dive into Otel due to its surface area being so big.

Re: The problem with OpenTelemetry

#52

Post by hinkley »

tnolet wrote: post #37

A recent example of OTel confusion. I could for the life of me not get the Python integration send traces to a collector. Same URL, same setup same API key as for Nodejs and Go. Turns out the Python SDK expect a URL encoded header, e.g. “Bearer%20somekey” whereas all other SDKs just accept a string with a whitespace. The whole split between HTTP, protobuf over HTTP and GRPC is also massively confusing.

The silent failure policy of OTEL makes flames shoot out of the top of my head.

We had to use wireshark to identify a super nasty bug in the “JavaScript” (but actually typescript despite being called opentelemetryjs) implementation.

And OTEL is largely unsuitable for short lived processes like CLIs, CI/CD. And I would wager the same holds for FaaS (Lambda).

In the end I prefer the network topology of StatsD, which is what we were migrating from. Let the collector do ALL of the bookkeeping instead of faffing about. OTEL is actively hostile to process-per-thread programming languages. If I had it to do over again I’d look at the StatsD->Prometheus integrations, and the StatsD extensions that support tagging.

Re: The problem with OpenTelemetry

#53

Post by EdSchouten »

> Its not a hard problem, [...]. At its core its structured events that carry two GUIDs along with them: a trace ID and a parent event ID. It is just building a tree.

I've always wondered, what's the point of the trace ID? What even is a trace?

- It could be a single database query that's invoked on a distributed database, giving you information about everything that went on inside the cluster processing that query.

- Or it could be all database calls made by a single page request on a web server.

- Or it could be a collection of page requests made by a single user as part of a shopping checkout process. Each page request could make many outgoing database calls.

Which of these three you should choose merely depends on what you want to visualize at a given point in time. My hope is that at some point we get a standard for tracing that does away with the notion of trace IDs. Just treat everything going on in the universe as a graph of inter-connected events.

Re: The problem with OpenTelemetry

#54

Post by tnolet »

hinkley wrote: post #52
tnolet wrote: post #37

A recent example of OTel confusion. I could for the life of me not get the Python integration send traces to a collector. Same URL, same setup same API key as for Nodejs and Go. Turns out the Python SDK expect a URL encoded header, e.g. “Bearer%20somekey” whereas all other SDKs just accept a string with a whitespace. The whole split between HTTP, protobuf over HTTP and GRPC is also massively confusing.

The silent failure policy of OTEL makes flames shoot out of the top of my head. We had to use wireshark to identify a super nasty bug in the “JavaScript” (but actually typescript despite being called opentelemetryjs) implementation. And OTEL is largely unsuitable for short lived processes like CLIs, CI/CD. And I would wager the same holds for FaaS (Lambda). In the end I prefer the network topology of StatsD, which is…

Yeah. And Otel has actually pretty nice debugging. You just need to set the right environment variable. But on prod it will blow up your logs

Re: The problem with OpenTelemetry

#55

Post by chipdart »

no_circuit wrote: post #39

IMO this boils down how one gets paid to understand or misunderstand something. A telemetry provider/founder is being commoditized by an open specification in which they do not participate in its development -- implied by the post saying the author doesn't know anyone on the spec committee(s). No surprise here. Of course implementing a spec from the provider point of view can be difficult. And also take a look at all…

> IMO this boils down how one gets paid to understand or misunderstand something.

I completely agree. The most charitable interpretation of this blog post is that the blogger genuinely fails go understand the basics of the problem domain, or worst case scenario they are trying to shitpost away the need for features that are well supported by a community-driven standard like OpenTelemetry.

Re: The problem with OpenTelemetry

#56

Post by serverlessmom »

An argument that OpenTelemetry is somehow 'too big' is an example of motivated reasoning. I can understand that A Guy Who Makes Money If You Use Sentry dislikes that people are using OTel libraries to solve similar problems.

Context propagation and distributed tracing are cool OTel features! But they are not the only thing OTel should be doing. OpenTelemetry instrumentation libraries can do a lot on their own, a friend of mine made massive savings in compute efficiency with the NodeJS OTel library: https://www.checklyhq.com/blog/coralogix-and-opentelemetry-o...

Re: The problem with OpenTelemetry

#57

Post by zeeg »

serverlessmom wrote: post #56

An argument that OpenTelemetry is somehow 'too big' is an example of motivated reasoning. I can understand that A Guy Who Makes Money If You Use Sentry dislikes that people are using OTel libraries to solve similar problems. Context propagation and distributed tracing are cool OTel features! But they are not the only thing OTel should be doing. OpenTelemetry instrumentation libraries can do a lot on their own, a frie…

Author here.

OpenTelemetry is not competitive to us (it doesn’t do what we do in plurality), and we specifically want to see the open tracing goals succeed.

I was pretty clear about that in the post though.

Re: The problem with OpenTelemetry

#58

Post by serverlessmom »

no_circuit wrote: post #39

IMO this boils down how one gets paid to understand or misunderstand something. A telemetry provider/founder is being commoditized by an open specification in which they do not participate in its development -- implied by the post saying the author doesn't know anyone on the spec committee(s). No surprise here. Of course implementing a spec from the provider point of view can be difficult. And also take a look at all…

I think that a number of Observability providers are looking at how they can add features and value to parts of monitoring that OTel effectively commoditizes. Thinking of the tail-based sampling implemented at Honeycomb for APM, or synthetic monitoring by my own team at Checkly.

"In 2015 Armin and I built a spec for Distributed Tracing. Its not a hard problem, it just requires an immense amount of coordination and effort." This to me feels like a nice glass of orange juice after brushing my teeth. The spec on DT is very easy, but the implementation is very very hard. The fact that OTel has nurtured a vast array of libraries to aid in context propagation is a huge acheivement, and saying 'This would all work fine if everyone everywhere adopted Sentry' is... laughable.

Totally outside the O11y space, OTel context propagation is an intensely useful feature because of how widespread it is. See Signadot implementing their smart test routing with OpenTelemetry: https://www.signadot.com/blog/scaling-environments-with-open...

Re: The problem with OpenTelemetry

#59

Post by brunoqc »

tnolet wrote: post #50
hahn-kev wrote: post #42

Earlier quoted context omitted.

Yeah but who wants to contribute to an SDK for a service that you need to pay for? That would be like if Oracle DB was open to contribution

Sentry is self hostable. https://develop.sentry.dev/self-hosted/

But not foss. It's using the BSL or FSL or whatever.

Re: The problem with OpenTelemetry

#60

Post by zeeg »

epgui wrote: post #40

Anyone else finding this very difficult to read? I’d really recommend feeding this through a grammar checker, because poor grammar betrays unclear thinking.

So you’re saying it makes my thinking more clear? :)

This is what happens when you use a tool designed for authoring code to also author content.

Post reply on HN