OpenTelemetry is interesting, On one side it is designed as the "commodity feeder" to number of proprietary backends as DataDog, on other hand we see good development of Open Source solutions as SigNoz and Coroot with good Otel support.
The problem with OpenTelemetry
81–90 of 178 posts
Re: The problem with OpenTelemetry
#82I understand what the author is saying, but vendor lock-in with closed-source observability platforms is a significant challenge, especially for large organizations. When you instrument hundreds or thousands of applications with a specific tool, like the Datadog Agent, disentangling from that tool becomes nearly impossible without a massive investment of engineering time. In the Platform Engineering professional serv…
Yeah, it's the primary reason we used it. If OpenTelemetry's raison d'être was simply to give Datadog a reason to not bullshit their customers on pricing, it would fulfill a major need in platform services.
Re: The problem with OpenTelemetry
#83Personally, I like OpenTelemetry, nice standardised approach. I just wished the vendors would have better support for the semantic conventions defined for a wide variety of traces. I quite like the idea of only need to change one small piece of the code to switch otel exporters instead of swapping out a vendor trace sdk. My main gripe with OpenTelemetry I don't fully understand what the exact difference is between (t…
> My main gripe with OpenTelemetry I don't fully understand what the exact difference is between (trace) events and log records. This is my main gripe too. I don't understand why {traces, logs, metrics} are not just different abstractions built on top of "events" (blobs of data your application ships off to some set of central locations). I don't understand why the opentelemetry collector forces me to re-implement th…
If you're using OTLP, SDKs only require you specify the endpoint once, the signal specific settings are for if you want to send them to different places.
The way you process/modify metrics vs logs vs traces are usually sufficiently different that there's not much point in having a unified event model if you're going to need a bunch of conditions to separate and process them differently. Of course, you can still use only one source (logs or events) and derive the other 2 from that, though that rarely scales well.
Plus, the backends that you can use to store/visualize the data usually are optimized for specific signals anyways.
Re: The problem with OpenTelemetry
#84It resonates. As an intern I had to add OTEL to a Python project and I had to spend a lot of time in the docs to understand the concepts and implementation. Also, the Python impl has a lot of global state that makes it hard to use properly imo.
> As an intern I had to ${DO_SOME_PROJECT} and I had to spend a lot of time in the docs to understand the concepts and implementation That sounds like every single run-of-the-mill internship.
That's fair but I'll say that the time and number of concepts you have to deal with before going into the code, per the docs; is quite big and I think the critic in the article is warranted.
Re: The problem with OpenTelemetry
#85There is a huge whole in using spans as they are specified. Without separating the start of a span from the end of a span you can never see things that never complete, fail hard enough to not close the span, or travel through queues. This is a compromise they made because typical storage systems for tracing aren't really good enough to stitch them all back together quickly. Everyone should be sending events and stitching it all together to create the view. But instead we get a least common denominator solution.
Re: The problem with OpenTelemetry
#86The main interest I've seen in OTel from Android engineers has been driven by concerns around vendor lock-in. Backend/devops in their organisations are typically using OTel tooling already & want to see all telemetry in one place.
From this perspective it doesn't matter if the OTel SDK comes bundled with a bunch of unnecessary code or version conflicts as is suggested in the article. The whole point is to regain control over telemetry & avoid paying $$$ to an ambivalent vendor.
FWIW, I don't think the OTel implementation for mobile is perfect - a lot of the code was originally written with backend JVM apps in mind & that can cause friction. However, I'm fairly optimistic those pain points will get fixed as more folks converge on this standard.
Disclaimer: I work at a Sentry competitor
Re: The problem with OpenTelemetry
#87I have been trying to find an equivalent for `tracing` first in Python and this week in TypeScript/JavaScript. At my work I created an internal post called "Better Python Logging? Tracing for Python?" that basically asks this question. OpenTelemetry was also what I looked at and since I have looked at other tooling.
It is hard to explain how convenient `tracing` is in Rust and why I sorely miss it elsewhere. The simple part of adding context to logs can be solved in a myriad of ways, yet all boil down to a similar "span-like" approach. I'm very interested in helping bring what `tracing` offers to other programming communities.
It very likely is worth having some people from the space involved, possibly from the tracing crate itself.
Re: The problem with OpenTelemetry
#88Earlier quoted context omitted.
No dog in the fight here, but… you're saying that one of the top guys at a major observability shop didn’t understand Open Telemetry, then that’s saying much more about OT than it does about his skills or efforts to understand. After all, his main point is that it’s complex and overengineered, which is the key takeaway for curious bystanders like me, whether every detail is technically correct or not. > it just reads…
OTel is very easy to add.. I've added it to several Go projects. For some frameworks like .NET you can do it automatically. The harder/more annoying part is setting up a viewer/collector like Jaeger. I've done that too but just in memory and it fills up quick.
For my small scale projects, Openobserve.ai has been super helpful. It ships as a single binary and (in non h/a setup) saves traces/logs/metrics to disk. I just set it up as a systems service and start sending telemetry via localhost. Code at https://github.com/bbkane/shovel_ansible/
Re: The problem with OpenTelemetry
#89Earlier quoted context omitted.
Our Linux devops engineer, who had not used Sentry before, set up a self-hosted Sentry in a day.
Yeah, it works for a time, but they don't support on-premise versions and they don't offer a Helm chart install, its all community based. I tried it for well over a year, and there are so many moving parts and so many "best guesses" from the community that we had to rip it out. There's a lot of components, sentry, sentry-relay, snuba, celery, redis, clickhouse, zookeeper (for clickhouse), kafka, zookeeper (for kafka)…
Day 1 vs day 2. That’s why the SaaS version exists.
Re: The problem with OpenTelemetry
#90OTel is flawed for sure, but I don't understand the stance against metrics and logs. Traces are inherently sampled unless you're lighting all your money on fire, or operating at so small a scale that these decisions have no real impact. There are kinds of metrics and logs which you always want to emit because they're mission-critical in some way. Is this a Sentry-specific thing? Does it just collapse these three kind…
I mean, when you're the one selling the gas to light that money on fire you have a vested interest in keeping it that way right? I do agree that logging and spans are very similar, but I disagree that logs are just spans because they aren't exactly the same. I also agree that you can collect all metrics from spans and, in fact, it might be a better way to tackle it. But it's just not feasible to do so monetarily so y…
Food for thought- the subjective nature of both of those is exactly why it shouldn’t be bundled.