Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The problem with OpenTelemetry

cra.mr

Post reply on HN
Print view View original

151–160 of 178 posts

Re: The problem with OpenTelemetry

#151

Post by pdimitar »

hinkley wrote: post #49
markl42 wrote: post #41

At the risk of hijacking the comments, I've been trying to use OTel recently to debug performance of a complex webpage with lots of async sibling spans, and finding it very very difficult to identify the critical path / bottlenecks. There's no causal relationships between sibling spans. I think in theory "span links" solves this, but afaict this is not a widely used feature in SDKs are UI viewers. (I wrote about this…

I was underwhelmed by the max size for spans before they get rejected. Our app was about an order of magnitude too complex for OTEL to handle. Reworking our code to support spans made our stack traces harder to read and in the end we turned the whole thing off anyway. Worse than doing nothing.

Sounds like a knob you can turn, from my practice at least.

Re: The problem with OpenTelemetry

#152

Post by pdimitar »

hinkley wrote: post #52
tnolet wrote: post #37

A recent example of OTel confusion. I could for the life of me not get the Python integration send traces to a collector. Same URL, same setup same API key as for Nodejs and Go. Turns out the Python SDK expect a URL encoded header, e.g. “Bearer%20somekey” whereas all other SDKs just accept a string with a whitespace. The whole split between HTTP, protobuf over HTTP and GRPC is also massively confusing.

The silent failure policy of OTEL makes flames shoot out of the top of my head. We had to use wireshark to identify a super nasty bug in the “JavaScript” (but actually typescript despite being called opentelemetryjs) implementation. And OTEL is largely unsuitable for short lived processes like CLIs, CI/CD. And I would wager the same holds for FaaS (Lambda). In the end I prefer the network topology of StatsD, which is…

> And OTEL is largely unsuitable for short lived processes like CLIs, CI/CD. And I would wager the same holds for FaaS (Lambda).

Not necessarily true f.ex. in one of my hobby Golang projects I found out that you can cleanly shutdown the OTel collector so it flushes its backlog of traces / metrics / logs so I was able to get telemetry reading even for CLI tool invocations that lasted 5-10 secs (connect to servers, get data, operate on it, put it someplace else, quit).

But now that you mention it, it would be nasty if that's not the default behavior indeed.

> OTEL is actively hostile to process-per-thread programming languages

Can you explain why, please?

Re: The problem with OpenTelemetry

#153

Post by pdimitar »

NeutralForest wrote: post #84
chipdart wrote: post #63

Earlier quoted context omitted.

> As an intern I had to ${DO_SOME_PROJECT} and I had to spend a lot of time in the docs to understand the concepts and implementation That sounds like every single run-of-the-mill internship.

That's fair but I'll say that the time and number of concepts you have to deal with before going into the code, per the docs; is quite big and I think the critic in the article is warranted.

Sounds like you have been overzealous about it. I just read an intro article or two -- maximum 15 minutes in total -- and I had an idea about trace spans vs. logging vs. metrics and I was on my merry way 1-2h later. We're talking the first time I needed OTel.

I am not humble-bragging at all, in fact I was quite dim back then (health problems reduced my focus and clarity of thought), I am saying that over-preparing can make everything seem untenable. So just don't.

Re: The problem with OpenTelemetry

#154

Post by pdimitar »

bigblind wrote: post #5
zaphar wrote: post #4

Earlier quoted context omitted.

Tracing requires keeping mappings for tracing identifiers per request. I don't know you do that without global state unless you want the tracing identifiers to pollute your own internal apis everywhere.

Many frameworks have the idea of a context" for this, that holds per-request state, following your reques through the system. Functions that don't care about the context just pass it on to whatever they call. I think Go was smart to make this concept part of the standard library, as it encouraged frameworks to adopt it as well.

Big agree on Golang. I had colleagues working with 5-6 JS microservices and they really wanted to keep span IDs so they can have sub-spans and unify semantic requests together (they also included background jobs processing) and they had to bring up a Redis container just to keep mappings of pieces of data every microservice has access to (e.g. order_id) and the actual span ID. It worked well in the end but it needed to be fully manually taken care of.

Re: The problem with OpenTelemetry

#155

Post by pdimitar »

bbkane wrote: post #88
cdelsolar wrote: post #31

Earlier quoted context omitted.

OTel is very easy to add.. I've added it to several Go projects. For some frameworks like .NET you can do it automatically. The harder/more annoying part is setting up a viewer/collector like Jaeger. I've done that too but just in memory and it fills up quick.

For my small scale projects, Openobserve.ai has been super helpful. It ships as a single binary and (in non h/a setup) saves traces/logs/metrics to disk. I just set it up as a systems service and start sending telemetry via localhost. Code at https://github.com/bbkane/shovel_ansible/

Same, I love OpenObserve and shill for it a lot. Maybe more advanced users have more use-cases and I am blind to it but all three main pillars of OTel (tracing spans, metrics, logs) it serves very well for my purposes.

Re: The problem with OpenTelemetry

#156

Post by pdimitar »

klabb3 wrote: post #26
drewbug01 wrote: post #17

As a contributor to (and consumer of) OpenTelemetry, I think critique and feedback is most welcome - and sorely needed. But this ain’t it. In the opening paragraphs the author dismisses the hardest parts of the problem (presumably because they are human problems, which engineers tend to ignore), and betrays a complete lack of interest in understanding why things ended up this way. It also seems they’ve completely mis…

No dog in the fight here, but… you're saying that one of the top guys at a major observability shop didn’t understand Open Telemetry, then that’s saying much more about OT than it does about his skills or efforts to understand. After all, his main point is that it’s complex and overengineered, which is the key takeaway for curious bystanders like me, whether every detail is technically correct or not. > it just reads…

> After all, his main point is that it’s complex and overengineered, which is the key takeaway for curious bystanders like me, whether every detail is technically correct or not.

This is a gross over-simplification that will leave you with a very skewed view of reality. As a programmer I only ever had to add a library, configure the OTLP endpoint details (host, port, URI, sometimes query parameters as well) and it was done.

It might be "complex and overengineered" if you want to contribute to the OTel libraries but as a programmer-user you are seeing practically none of it. And I would also challenge the "complex and overengineered" part but for now I am not informed enough to do it.

Re: The problem with OpenTelemetry

#157

Post by pdimitar »

aserafini wrote: post #45
the_mitsuhiko wrote: post #29

Earlier quoted context omitted.

I believe on the infrastructure side that might be correct. Within applications that doesn’t match my experience. In many cases the concurrent nature of servers makes it impossible to repro issues and narrow down the problem without tracing or trace aware logs.

With only sampled traces though it’s very hard to understand the impact of the problem. There are some bad traces but is it affecting 5%, 10% or 90% of your customers. Metrics shine there.

Strange example, you'd think you want to fix this as quickly as humanly possible, no?

Also we don't sample traces, it's a fire hose of data aimed at the OTel collector. We do archive them / move them to colder and cheaper storage after a little time though, and we found that a viable money-saving strategy and a good balance overall.

Re: The problem with OpenTelemetry

#158

Post by pdimitar »

ris wrote: post #95

1. The main reason I want to use otel is so I can have one sidecar for my observability, not three, each with subtly different quirks and expectations. (also the associated collection/aggregation infrastructure) 2. I honestly think the main reason otel appears so complex is the existing resources that attempt to explain the various concepts around it do a poor job and are very hand-wavey. You know the main thing that…

Your point 2 would make for a very interesting blog post worthy of HN submitting. :)

Re: The problem with OpenTelemetry

#159

Post by pdimitar »

dan-allen wrote: post #107

I keep checking in on OpenTelemetry every few months to see if the bits we need are stable yet. There’s been very little progress on the things we’re waiting for. I don’t follow closely enough to comment on possible causes. What I do know is that the surface area of code and infrastructure that telemetry touches means adopting something unfinished is a big leap of faith.

IMO you might be looking at the wrong signals, OTel is quite successful today and I had zero breakages or compatibility problems for at least 2 years at this point.

Re: The problem with OpenTelemetry

#160

Post by pdimitar »

zeeg wrote: post #79
zellyn wrote: post #76

Are they basically just saying that the OpenTelemetry client APIs should be split from the rest of the pieces of the project, and versioned super conservatively? The simple API they describe is basically there in OTel. The API is larger, because it also does quite a few other things (personally, I think (W3C) Baggage is important too), but as a library author I should need only the client APIs to write to. When imple…

It’s a bit deeper than that. The SDKs that library authors implement need to be extemely minimal. The collection libraries that vendors implement based on imo should also be minimal. OTLP imo doesn’t even need to be part of the spec. But minimal would also mean focusing on solving fewer problems as a whole. Eg OpenTracing plus OpenMetrics plus OpenLogs. I only need one of those things.

Well, on OTLP they seem to agree with you: https://opentelemetry.io/blog/2023/otel-arrow/
Post reply on HN