Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenTelemetry: Escape Hatch from the Observability Cartel

oneuptime.com

Post reply on HN
Print view View original

61–70 of 73 posts

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#61

Post by siegecraft »

arccy wrote: post #38
jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

really? for a lot of us the datadog query language for dashboards is absolute trash. getting it to do what you want sits somewhere between difficult to impossible. i guess it depends on what you're used to

what do you use instead and what does it give you that you can't get from datadog? as someone who's been locked into datadog for the past few years i'm wondering what i've been missing.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#62

Post by pan69 »

secondcoming wrote: post #36
mohammadv184 wrote: post #27

There is an un-marketed reality: OTel is not simple. The learning curve is steep, the documentation can be a maze of specs, and the SDKs (especially for metrics and logs) can feel over-engineered. You will get frustrated.

The C++ SDK is a masterpiece of over-engineering.

So is the JavaScript/TypeScript one. Very steep learning curve, very fragmented but clearly also very powerful once you know how to use it.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#63

Post by schainks »

ramon156 wrote: post #24

OTel has been slow as hell since release. we tried using it for tracing last week and the app was significantly slower. We just use prometheus+grafana now. Yes, this technically also slows the app down, but OTel was unbearably slow. I'm sure im doing a million and one things wrong, but i can't be arsed to set something up just to see some performance metrics. Deadlocks can be found using transaction metrics, that's a…

Say more about what you did to instrument? I am willing to bet architectural choices are that slowness, not OTel itself.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#64

Post by greatgib »

PunchyHamster wrote: post #19
greatgib wrote: post #16

I was always turned down to use more Otel by how verbose it is and how heavy are the telemetry payload compared to simple adhoc alternatives. Am I wrong?

It is designed to get insights into dozen(s) connected systems. It will always be overkill for just an app or two talking with eachother... till you grow, then it won't be overkill any more. But still might be worth getting into it on smaller apps just thanks to wealth of tools available.

But my point is, especially for a big system, the performance impact of just the telemetry should be huge.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#65

Post by makeavish »

jitl wrote: post #57
voxic11 wrote: post #46

Earlier quoted context omitted.

Have you looked at https://signoz.io/ ? Its the closest I have tried, but I still wouldn't put it quite at the same level of usability.

Doesn't seem like this can visualize over data stored in Honeycomb like Grafana can :(

It’s a full fledged tool like DataDog which is opensource and can be selfhosted. You can replace honeycomb with it. What features do you think it’s missing compared to honeycomb?

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#66

Post by morningspace »

boilerupnc wrote: post #51
jitl wrote: post #14

Does any OpenTelemetry vendor have a dashboard / graph product in the same level of usability as Datadog? Honeycomb is decent at what it has, but very limited offerings for dashboard. Coming from Datadog, Grafana is such a bad experience I want to cry every time I try to build out a service dashboard. So much more friction to get anything done like adding transform functions / operators, do smoothing or extrapolation…

Give IBM Instana a look. It has great otel support. Nice usability. [0][1] Free sandbox for an email [2] [0] https://www.ibm.com/products/instana/opentelemetry [1] https://github.com/instana/instana-otel-collector [2] https://play-with.instana.io/#/home [disclaimer: I'm an IBMer]

> Can this visualize over data stored in Honeycomb like Grafana can?

IIUC, Grafana connects directly to Honeycomb via its API to visualize data without storing it. Instana, on the other hand, is a bit different. It needs telemetry data to be ingested into backend before it can be visualized in UI. With Honeycomb, this could be possible if the data can be exported from Honeycomb to Instana.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#68

Post by kchoudhu »

rixed wrote: post #28
kchoudhu wrote: post #13

OpenTelemetry won observability mindshare, but it is entirely the wrong architectural choice: by buying into its ethos your code is held hostage by the least stable otel monitoring library for your dependencies. Sadly, there was always an alternative that no one took: dtrace. Add USDTs to your code and then monitor progress by instrumenting it externally, sending the resulting traces to wherever you want. My sincere…

I'm not familiar with usdt, but my understanding is that it covers only the lower level of Opentelemetry. Is there an equivalent of OTLP for dtrace?

I cannot believe it can be too hard to write something that takes the output of dtrace and shoves it into a span collector.

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#69

Post by cstrahan »

carefulfungi wrote: post #5
csomar wrote: post #4

Earlier quoted context omitted.

You still have to do that work yourself. I am using honeycomb (the free tier) but their pricing makes little sense. Their margins must be something like x100.

Having shipped free-tier observability products, your comment that you aren't paying them but think their margins are 100x is a perfect irony.

It sure sounds like the problem is subsidizing the free tier by making the paid tiers excessively expensive. That in turn drives people to squeeze as much out of the free tier as they can, complain online, and jump ship to either a different service or self hosting.

So... maybe don't do that?

Re: OpenTelemetry: Escape Hatch from the Observability Cartel

#70

Post by nijave »

blyry wrote: post #47
nijave wrote: post #29

Earlier quoted context omitted.

I thought New Relic was pretty good but their per seat license cost is super high They have a SQL like query language that I think can do most of what you're describing

We migrated from newrelic to datadog (for cost reasons LMAO) a while back and I miss NRQL every single day I'm building a dashboard. I enjoy having everything instrumented and in one spot, it's super powerful, but I am currently advocating for self hosting loki so that we can have debug+ level logs across all environments for a much much lower cost. Datadog is really good at identifying anomalies, but the cost for lo…

Datadog log ingest isn't too expensive which will still enrich the logs and can dump to S3 (log archive) where you can use Athena/Trino

Log indexing is $$$, though for sure

Curious on Loki cost. When I priced out ELK at a smaller company it didn't come in much cheaper than $0.50/Gi everyone seems to charge (30 day retention, 2 shards, object storage backups). Back when I worked at JPMC, their internal service was also billed right around there.

Post reply on HN