Live data from Hacker News

The problem with OpenTelemetry

cra.mr

161–170 of 178 posts

Re: The problem with OpenTelemetry

#161
post #134

Earlier quoted context omitted.

Right, the point I’m making is logs, metrics, traces, these concepts are views of data, with a pretty hazy relationship to the shape of the data itself or the handling requirements. Any assumption you make about them as a category (logs are unstructured, traces are sampled, metrics can be aggregated) is wrong nearly as much as it’s right.

> Right, the point I’m making is logs, metrics, traces, these concepts are views of data (...) Not really. Logs are fundamentally different than operational metrics, which are fundamentally different than business/behavioral metrics, which are fundamentally different than traces, etc etc etc. This is not a matter of "view". It's the result of completely different system requirements. They are emitted differently, the…

> Click stream metrics mix traits of tracing with logging and metrics

This sounds like you are admitting my point? My point is not “there is no difference between anything” my point is that the 3 buckets of “metrics, logs, traces” are neither all-encompassing in terms of types of telemetry one might feasibly want to emit, nor are they mutually exclusive. Here is perhaps a better writeup of what I mean

https://open.substack.com/pub/isburmistrov/p/all-you-need-is...

Re: The problem with OpenTelemetry

#162
post #103
post #100

Earlier quoted context omitted.

that just sounds like a branding problem though... OTLP has been quite useful especially in metrics to get a format that doesn't really have any sacrifices/limitations compared to all the other protocols.

It is! But to prove your point, OTLP is actually just the transport protocol (Open Telemetry Transport Protocol). Its one of _so many things_ its trying to address. All of those things might be probems, but not everyone has those same problems (vendors, customers, and lib authors), and bundling them all into one umbrella just screams for me. I actually have no need for a standard metrics implementation, just as an ex…

I read all your comments in this thread and I am still confused as to what your actual problem is.

Would it help you if all language-specific OTel libraries had 3 parts: metrics, log, traces? Namely you want finer-grained opt-in approach for the programmer-users? Or is it something else you have a problem with?

Saying OTel is a failure of a design committee process is to me blowing hot air; you are telling us your conclusion and I personally care exactly zero about your conclusion. I want to see your process of arriving at the said conclusion. And so far neither the OP nor your comments here gave me almost any enlightenment in this regard.

Are you open to clarifying further?

Re: The problem with OpenTelemetry

#163
post #104
post #99

Earlier quoted context omitted.

I think they meant "an event ID and a parent event ID".

I actually meant trace ID and parent event ID (and ID was inferred). Parent comment is correct in that trace ID isnt technically needed, and is in fact quite controversial. Its an implementation level protocol optimization though, and unfortunately not an objective one. It creates an arbitrary grouping of these annotations - which is entirely subjective, and the spec struggles to reconcile - but its primarily because…

I am absolutely not seeing the problem you describe, I appreciate the ability to have a single ID for stuff that might go through several microservices. Helps us discover all data related to a single request-like entity in case of trouble. It already helped us trace semi-failed complex processes related to separate customers and we were able to manually fix things in a matter of minutes.

As said in other replies to you, I am still not at all clear what your issue with OTel is.

Re: The problem with OpenTelemetry

#164
post #103

Earlier quoted context omitted.

It is! But to prove your point, OTLP is actually just the transport protocol (Open Telemetry Transport Protocol). Its one of _so many things_ its trying to address. All of those things might be probems, but not everyone has those same problems (vendors, customers, and lib authors), and bundling them all into one umbrella just screams for me. I actually have no need for a standard metrics implementation, just as an ex…

I read all your comments in this thread and I am still confused as to what your actual problem is. Would it help you if all language-specific OTel libraries had 3 parts: metrics, log, traces? Namely you want finer-grained opt-in approach for the programmer-users? Or is it something else you have a problem with? Saying OTel is a failure of a design committee process is to me blowing hot air; you are telling us your co…

I want one fundamental thing: the ability to ask library authors to implement span annotations in their projects. Today that ask comes with way too much baggage. The SDKs are often extremely complex ("bloated", other's words, not mine), and on top of that, most of these library authors don't use or care about the standard. The latter can be fixed w/ funding, the former is what I'm concerned complaining about.

What I mean by that in practical terms is very easily articulated when we look at something like bundling of logs. We've had standardzed formats, adapters, and transports for logging for decades. Could they be better? Sure. Aint no future though where someone builds an SDK and no one ever again innovates or has an alternative path to achieving it. The same is true for everything, logs are just an an easy thing to pick on.

Take that a little further - why do I need a logging SDK bundled with a tracing SDK? I mean that very literally - why _must_ they be bundled? What do we get out of it?

You can argue that "just dont use the other parts", but thats an academic argument at best. In practice that just means you've got an overloaded SDK with a bunch of bias associated with it - in this case (in my opinion) a bunch of companies who don't really innovative pushing legacy telemetry concepts on the masses. That might not sound like a problem, but a developer can barely make sense of the docs, and even at the most basic level, I shouldnt need to make an argument about software branching complexity to other software engineers. They're simply trying to do too many things.

What that ultimately boils down to me though is: what problem is it solving? Tracing is a serious value add to _most_ stacks these days that requires immense amount of coordination to solve. It's not maintainable (easily) through patching other's code, which goes back to why I personally agree w/ OpenTracing's original pitch. However, why, with that goal unsolved, did we attach a bunch of other loosely-but-mostly unrelated problems to the spec? At the very least, problems that dont apply universally to the same audience. Why did we need a spec that bundled all of these problems, and continues to try to bundle more?

So I go back to the core problem: we want universal span annotations implemented across the ecosystem. I dont see what OTel is doing as the most effective way to achieve that goal, and I dont see the other goals they're trying to solve for as ones that are actually that important to most developers (and quite frankly, I could not tell you what many of those goals even are).

A lot of the development of OTel looks like two things:

1) Startups wanting easy access to telemetry, often without any product differention, thus, stakeholders who I dont find totally relevant to the conversation. I'm sure that comes off as me being an asshole, but its how I feel.

2) Big vendors trying to push consolidation on the customers, all competing with the exact same products (think Datadog and all its copycats, again often with no product innovation). This isnt totally a problem except it amounts to them pushing fragmented legacy concerns (such as outdated logging concepts) downstream.

All I want is great quality of data for both developers of libraries and developers of applications - those are the customers. I dont see those customers being serviced the best they could be, and I dont see the goals being met _because_ of the distraction, lack of focus, and as far as I can tell, lack of vision of the project.

I think anyone who's working on OTel, if they genuinely had the best interests of the developers in mind, would be hard pressed to answer why Sentry's support isn't an extremely desirable thing given our market reach. The people that care about the project want us involved (and if you're reading this, thank you for constantly pushing us), and I want us to also be involved. So far though we're constantly struggling to actually get an uninstrusive implementation in place that can work with our product, and what I'm asking for would solve for that, but to me looks like a cultural problem and not a technical problem...

The issue is we currently dont see a great incentive to fund a bunch of piecemeal standards that arent relevant to our product, and many library authors are not going to naturally invest into this. Even more say, many of these authors that I've talked to have no excitement what so ever about the standard, or worse, an active distaste. People can say they're wrong, but frankly, that doesnt matter. You dont succeed by thinking someone else is wrong, you succeed by building what your customers wnat.

That is why I would like to see the project focus on problems that people actually have, and do it in a way that doesn't create tradeoffs for developers. To me those problems are the ones that aren't achievable without this level of coordination. They are not shimming another log or metrics collector in place. Those things might be relevant to some people, and thats fine, but we dont need one be-all-end-all project to encompass all sorts of fuzzy semi-related problems.

I may or may not be making sense here, and I'm happy to chat about it more, though HN is probably not a good venue for that.

Re: The problem with OpenTelemetry

#165
post #96

Earlier quoted context omitted.

With only sampled traces though it’s very hard to understand the impact of the problem. There are some bad traces but is it affecting 5%, 10% or 90% of your customers. Metrics shine there.

Whether it is affecting 5% or 10% of your customers, if it is erroring at that rate you are going to want to find the root cause ASAP. Traces let you do that, whereas the precise number does nothing. I am a big supporter of metrics but I don't see this as the use case at all.

(not your OP) This is true, but I find that metrics are useful whether something is going wrong or not (metrics that show 100% success are useful in determining baselines and what "normal" is), whereas collecting traces _when nothing is going wrong_ is not useful -- it's just taking up space and ingress, and thus costing me money.

My typical approach in the past has been to use metrics to determine when something is going wrong, then enable either tracing or logs (usually logs) to determine exactly what is breaking. For a dev or team that is highly connected to their software, simply knowing what was recently released is enough to zero in on problems without relying upon tracing.

Traces can be useful, but they're expensive relative to metrics, even if sampled at a very low rate.

Re: The problem with OpenTelemetry

#166
post #136

Earlier quoted context omitted.

Personally I could never get into k8s. For most of my use-cases, simply getting a VPS, or having a Dockerfile is enough. Most apps can run fine for millions or hundreds of thousands of user sessions on a $5-$50 VPS. People prematurely optimize for scale, adding a lot of complexity that only makes development slower, and by having more moving parts, there are more things that can break. Start simple. Scaling is mostly…

Yep, strong agree here, and indeed the modern hardware can achieve mind-blowing stuff. I have used a $250 laptop (i5 CPU that's now 8-9 years old, 16GB RAM, SATA SSD) to stress test some of my Rust apps and I could achieve ~250K requests per second. Sure that doesn't apply to all tech (f.ex. my favorite Elixir is much slower than this) but for most intents and purposes a plain old VPS with good backups is quite enoug…

A bit offtopic, but do you recommend any load balancer?

I could have another VPS as a load balancer, but what if goes down? The load balancing services seem quite expensive, as they charge per request.

Re: The problem with OpenTelemetry

#167
post #166

Earlier quoted context omitted.

Yep, strong agree here, and indeed the modern hardware can achieve mind-blowing stuff. I have used a $250 laptop (i5 CPU that's now 8-9 years old, 16GB RAM, SATA SSD) to stress test some of my Rust apps and I could achieve ~250K requests per second. Sure that doesn't apply to all tech (f.ex. my favorite Elixir is much slower than this) but for most intents and purposes a plain old VPS with good backups is quite enoug…

A bit offtopic, but do you recommend any load balancer? I could have another VPS as a load balancer, but what if goes down? The load balancing services seem quite expensive, as they charge per request.

One solution I was thinking of, is to load balance using DNS, or at least to have failure recovery using DNS (one server goes down, update the DNS to point to the other server). I am fine with a small downtime.

Re: The problem with OpenTelemetry

#168
post #166

Earlier quoted context omitted.

Yep, strong agree here, and indeed the modern hardware can achieve mind-blowing stuff. I have used a $250 laptop (i5 CPU that's now 8-9 years old, 16GB RAM, SATA SSD) to stress test some of my Rust apps and I could achieve ~250K requests per second. Sure that doesn't apply to all tech (f.ex. my favorite Elixir is much slower than this) but for most intents and purposes a plain old VPS with good backups is quite enoug…

A bit offtopic, but do you recommend any load balancer? I could have another VPS as a load balancer, but what if goes down? The load balancing services seem quite expensive, as they charge per request.

Ah, I meant use your hosting provider's load balancer + use their VPS-es. I think Digital Ocean allows this, AWS and GCP as well.

That way you're outsourcing the problem of "but what if the load balancer itself falls down?".

Re: The problem with OpenTelemetry

#169
I think there are two separate perspectives. For developers Open Telemetry is a clear win - high-quality vendor agnostic instrumentation backed by a reputable orgs. I instrumented with traces many business critical repos at my company (major customer support SaaS) with OTEL in Ruby, Python, JS. Not once was I confused/blocked/distracted by the presence of logs/metrics in the spec. However, can’t say much from the observability vendor perspective trying to be fully compatible with OTEL spec including metrics/logs. Article mentions customers having issues with using tracing instrumentation - it would’ve been great to back this up with corresponding github issues explaining the problems. Based on the presented JS snippet (just my guess) maybe the issue is with async code where the “span.operation” span gets immediately closed w/o waiting for the doTheThing()? Yeah - that’s tricky in JS given its async primitives. We ended up just maintaining a global reference to the currently active span and patching some OTEL packages to respect that. FWIW Sentry JS instrumentation IS really good and practical. Would have been great if Sentry could donate/contribute/influence to OTEL JS SIG with specific improvements - would be a win-win. As much as I hate DataCanine pricing they did effectively donated their Ruby tracing instrumentation to OTEL which I think is one of the best ones out there.

Re: The problem with OpenTelemetry

#170
post #166

Earlier quoted context omitted.

A bit offtopic, but do you recommend any load balancer? I could have another VPS as a load balancer, but what if goes down? The load balancing services seem quite expensive, as they charge per request.

Ah, I meant use your hosting provider's load balancer + use their VPS-es. I think Digital Ocean allows this, AWS and GCP as well. That way you're outsourcing the problem of "but what if the load balancer itself falls down?".

I usually want to load-balance between different hosting providers (so if AWS goes down, I still have DO running). From what I've seen, most of them only provide load-balancing within their local networks.
Post reply on HN