Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OTel isn’t going well

matduggan.com

Post reply on HN
Print view View original

121–130 of 132 posts

Re: OTel isn’t going well

#121

Post by thiht »

osener wrote: post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

The official Go SDK is a nightmare to use and understand. Just the "getting started"[1] requires adding 10 imports.

[1]: https://opentelemetry.io/docs/languages/go/getting-started/#...

Re: OTel isn’t going well

#122

Post by bilalq »

cyberax wrote: post #118
bilalq wrote: post #93

Earlier quoted context omitted.

If you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you n…

Why do you even _need_ these "layers"? It's a simple RPC protocol that submits data tagged with Span and Trace IDs. That's really all there is to it. You can just submit it directly, without involving any layers.

Because blocking on OTel data to flush before sending a response back is often unacceptable. The layers run a standalone process using the Lambda extension API so they can keep running after your function has responded to a request.

Re: OTel isn’t going well

#123

Post by gcbirzan »

cyberax wrote: post #13

I disagree. I'm an observability geek, and OTel is... fine. It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements: 1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" quer…

We did the same for 1, we have a service that picks up messages from a queue and drops 95% of them. In Python, at least, it was really really ugly. We had the alternative of using refinery, since we're on honeycomb, but didn't want to get stuck with something that only works with one vendor.

I don't understand 2, though, that should be easy? You have access to the span and you can set its status code to 0. With the Python SDK that was trivial for us to do.

The one thing I really hate about the Python SDK is that there's no simple way to say "attach this attribute to the top level span in this app". That way, when you get the user id further down in your app, you can just annotate the top level span with it. Thankfully, honeycomb has any, any2, root, parent (that allows you to filter on any span in the trace, or on the root or parent) but that's slow and again, vendor specific.

Re: OTel isn’t going well

#124

Post by csujoy »

Havoc wrote: post #64

I find the entire observability space to quite a poor experience, at least in the self-hosted space. Tried both grafana route and signoz and neither seems particularly pleasant

IMO Grafana is definitely the best o11y tool when you are just starting out. But it isn't built for scale. When you have to analyze petabytes of logs, something custom built for this like CtrlB might be more helpful.

Re: OTel isn’t going well

#125

Post by ruhani_grover »

Havoc wrote: post #112
nunez wrote: post #84

Earlier quoted context omitted.

What about the experience did you find lacking?

That would be an essay, but in short: Grafana is fine to get to a selfhosted basic install. But once you try to actually connect logs, metrics, traces in selfhosted context and perhaps sprinkle some otel in...that sht gets out of hand very fast. It's modular in a way that seems like a win but once you start connecting stuff it starts adding complexity not ease. Signoz...still pretty early in exploring this and so far…

[dead]

Re: OTel isn’t going well

#126

Post by growse »

kalkin wrote: post #71
osener wrote: post #32

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away. It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions”…

The article assumes the issue with OTel is slow feature development, which isn't my experience at all. The issue I've had is that the SDKs have terrible performance overhead for instrumentation and are, as you say, highly resistant to integrating the output of better performing (or just preexisting) instrumentation. In Python and Ruby, at least, the CPU cost of all the mandatory abstraction is way too high.

What I find confusing about this is that otel is two things.

1. A spec 2. A ref implementation

Similar to other projects (e.g. python), if there's complaints about (2), that should trigger an ecosystem of alternative implementations that are guaranteed to be compatible because of (1).

I suspect there's actually quite a few private, separate otel implementations. Maybe these just aren't being contributed as oss?

Re: OTel isn’t going well

#127

Post by growse »

PunchyHamster wrote: post #76
twic wrote: post #58

Earlier quoted context omitted.

Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. A trace is a period of execution between two events. You could record a trace as a pair of log entries, or one log entry at the end. You can then rec…

> Logs and metrics are both derived from events. A log takes the whole event and records it somewhere. A metric takes some numeric value from the event, aggregates it over time, and records it periodically. You can reconstruct a metric from logs for the underlying events. No, metric is just value. Some are derived from events (like histogram/rate of given event duration) but others are wholly independent (like return…

A clock ticking every second is generating an event every second.

If you sample the CPU usage at 1Hz, the metric is attached to the tick event.

Re: OTel isn’t going well

#128

Post by dengolius »

nlitened wrote: post #44
tete wrote: post #39

OpenTelemtry is the perfect example of an overengineered mess. While I usually think that at least having some standard that people agree on I think OpenTelemtry should be dropped. A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards and a lot better both in terms of less added complexity and the results you get. I think OpenTelemetry turne…

I agree overall, however: > A lot of the less popular alternatives (just going with Prometheus, Victoriametrics, etc) are de-facto competing smaller standards By all metrics (hah), Prometheus is the more popular solution and is the de-facto standard, as far as I know.

What about InfluxDB? Speaking of standards, it’s worth noting that this system was created early on, when microservices were emerging as a concept. Another reason this “de facto standard” emerged is that there were almost no alternatives. This is what we need to understand about standards and who promotes them. But technology doesn’t stand still, and I wouldn’t call Prometheus the standard right now, because OpenTelemetry is already starting to be referred to as the standard in the field of observability. Some people like it, some don’t, but Zabbix and Nagios are still going strong; for some, they remain the standard for monitoring for a variety of personal reasons.

Re: OTel isn’t going well

#129

Post by dengolius »

pphysch wrote: post #100
czhu12 wrote: post #95

It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host. Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.

There is, it's called VictoriaMetrics/Logs/Traces. https://victoriametrics.com/

Would you like to share your success story using this software?

Re: OTel isn’t going well

#130

Post by dengolius »

Havoc wrote: post #112
nunez wrote: post #84

Earlier quoted context omitted.

What about the experience did you find lacking?

That would be an essay, but in short: Grafana is fine to get to a selfhosted basic install. But once you try to actually connect logs, metrics, traces in selfhosted context and perhaps sprinkle some otel in...that sht gets out of hand very fast. It's modular in a way that seems like a win but once you start connecting stuff it starts adding complexity not ease. Signoz...still pretty early in exploring this and so far…

> Grafana is fine to get to a selfhosted basic install. But once you try to actually connect logs, metrics, traces in selfhosted context and perhaps sprinkle some otel in...that sht gets out of hand very fast. It's modular in a way that seems like a win but once you start connecting stuff it starts adding complexity not easy.

Maybe because some open-source software has limitations? That seems reasonable to me.

Post reply on HN