Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenTelemetry: Escape Hatch from the Observability Cartel

oneuptime.com

Post reply on HN
Print view View original

51–60 of 73 posts

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#51

Post by boilerupnc »

jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

Give IBM Instana a look. It has great otel support. Nice usability. [0][1] Free sandbox for an email [2]

[0] https://www.ibm.com/products/instana/opentelemetry

[1] https://github.com/instana/instana-otel-collector

[2] https://play-with.instana.io/#/home

[disclaimer: I'm an IBMer]

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#52

Post by yread »

nijave wrote: post #29
jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

I thought New Relic was pretty good but their per seat license cost is super high They have a SQL like query language that I think can do most of what you're describing

I use New Relic and I'm not sure I would recommend it. Sure it was easy to install on all the server but now I keep getting alerts every months (on like the 2nd day) that I ran out of some "quota" of 100GB. I thought that's impossible as I almost don't use it but apparently they (used to) send by default list of running processes every second and that gunks things up.

Also I setup alerts - for >50% CPU or >90% disk full. I do get an alert but it doesn't say which volume or how full is it - what was the actual value that triggered sending the alert. WTF.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#53

Post by _boffin_ »

everfrustrated wrote: post #50
jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

As someone used to Datadog I can't find anything that comes close. Datadog makes it so easy to throw adhoc data up on a screen and explore a hypothesis from there. Using Grafana or New Relic feels like going from a 30" screen to a 14" laptop - everything is just harder and more frustrating in ways that are hard to articulate.

Elastic is my choice. Don’t like grafana and dd felt like it was holding your hand while telling you to give them your money as they punch you repeatedly.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#54

Post by r3tr0 »

OTel's real bottleneck isn't the spec. It's the fact that it requires you to instrument the app itself. That couples your performance to the maturity of the least stable SDK you depend on.

eBPF solves this by reversing the model: instrument the system, not the application. Turn it on / off dynamically, zero redeploys, minimal overhead.

The missing piece is accessibility. Kernel-level observability exists; "normal engineers can use it" and good DX does not.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#55

Post by jitl »

mvf4z7 wrote: post #40
jitl wrote: post #21

Earlier quoted context omitted.

The JavaScript Otel packages are implemented with 34 layers of extra abstraction. We wrote our own implementation of tracing for Cloudflare Workers and it performs much better with 0 layers of abstraction. I’ve seen a few other services switching over to our lightweight tracer. The emitted JSON is still chunky but removing all the incidental complexity helped a lot.

Can you share a link to your implementation?

It's private code in our monorepo

EDIT: we actually have two. The one we use for Node, the author plans to open source it eventually. That one is drop in replacement for Span and Trace classes and Just Works with upstream Otel. Main blocker is that we have some patch-package to fix other performance issues with upstream, and need to make our stuff work with non-patched upstream.

The one we use for Workers is more janky and doesn’t make sense to open source. It’s like 100 total LoC but doesn’t have compatibility with existing Otel libraries.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#56

Post by jitl »

boilerupnc wrote: post #51
jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

Give IBM Instana a look. It has great otel support. Nice usability. [0][1] Free sandbox for an email [2] [0] https://www.ibm.com/products/instana/opentelemetry [1] https://github.com/instana/instana-otel-collector [2] https://play-with.instana.io/#/home [disclaimer: I'm an IBMer]

Can this visualize over data stored in Honeycomb like Grafana can?

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#57

Post by jitl »

voxic11 wrote: post #46
jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

Have you looked at https://signoz.io/ ? Its the closest I have tried, but I still wouldn't put it quite at the same level of usability.

Doesn't seem like this can visualize over data stored in Honeycomb like Grafana can :(

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#58

Post by phillipcarter »

ramon156 wrote: post #24

OTel has been slow as hell since release. we tried using it for tracing last week and the app was significantly slower. We just use prometheus+grafana now. Yes, this technically also slows the app down, but OTel was unbearably slow. I'm sure im doing a million and one things wrong, but i can't be arsed to set something up just to see some performance metrics. Deadlocks can be found using transaction metrics, that's a…

Would be curious about this in more detail since I’ve not normally seen a JS app be significantly slower due to adding autoinstrumentation over the years. There’s obviously an overhead but aside from the occasional bug I’ve never seen it be significant to impact user experiences or cost to serve.

That said if your goal is basic performance metrics and nothing more, then tracing is overkill. Don’t even need an SDK, just monitor the compute nodes async with some basic system metrics. But if your goal is to narrow down behaviors within your app on a per-request basis there really is no way around tracing if you value your sanity.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#59

Post by ok_dad »

kchoudhu wrote: post #45
PunchyHamster wrote: post #23

Earlier quoted context omitted.

dtrace was meant for entirely different use, and it's not a replacement for otel Otel was made to basically track the request execution (and anything that request triggers) across multiple apps at once, not to instrument an app to find slow points

We should be using USDTs to emit trace ids that can be consumed by dtrace and shoved into whatever backend we want for tracing.

What’s a USDT? All I can find on Google is crypto garbage.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#60

Post by tofflos »

ok_dad wrote: post #59
kchoudhu wrote: post #45

Earlier quoted context omitted.

We should be using USDTs to emit trace ids that can be consumed by dtrace and shoved into whatever backend we want for tracing.

What’s a USDT? All I can find on Google is crypto garbage.

https://illumos.org/books/dtrace/chp-usdt.html#chp-usdt
Post reply on HN