Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OTel isn’t going well

matduggan.com

Post reply on HN
Print view View original

111–120 of 123 posts

Re: OTel isn’t going well

#111

Post by arccy »

czhu12 wrote: post #95

It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host. Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.

clickstack https://clickhouse.com/clickstack

is the closest i've seen to the datadog experience

Re: OTel isn’t going well

#112

Post by Havoc »

nunez wrote: post #84
Havoc wrote: post #64

I find the entire observability space to quite a poor experience, at least in the self-hosted space. Tried both grafana route and signoz and neither seems particularly pleasant

What about the experience did you find lacking?

That would be an essay, but in short:

Grafana is fine to get to a selfhosted basic install. But once you try to actually connect logs, metrics, traces in selfhosted context and perhaps sprinkle some otel in...that sht gets out of hand very fast. It's modular in a way that seems like a win but once you start connecting stuff it starts adding complexity not ease.

Signoz...still pretty early in exploring this and so far it's acceptable, but it too relies on a mix of query languages incl the competitors promql so some panels support it other seem to not to?

The entire thing just seems bewildering to me. Not gonna say "why is this so hard" because I genuinely thing smart people are genuinely trying here...but there result just isn't great.

Re: OTel isn’t going well

#113

Post by psadri »

I looked at the OTel schema generated for recording a single numerical metric. It was like 12 or 13 meta fields in addition to the actual metric fields like timestamp, metric and value.

OTel looks like something designed by a committee of committees, funded by someone who is in the business of selling cloud storage/data warehousing services.

Re: OTel isn’t going well

#114

Post by rtpg »

pphysch wrote: post #100
czhu12 wrote: post #95

It really never grokked with me why there isn't just "open source Datadog" that can be installed and used. End to end, stateful, that we can just self host. Our team tried to set up open telemetry to replace Datadog and got totally crushed in complexity. The model of having Open Telemetry just be for standardizing & exporting to other backends, needing glue for each part of the setup was nuts.

There is, it's called VictoriaMetrics/Logs/Traces. https://victoriametrics.com/

We use victoriametrics, but I believe that's just the collector side of it cuz we also query into it with grafana.

Datadog isn't just a collector, but the whole querying UI as well, right?

Re: OTel isn’t going well

#115

Post by rtpg »

cyberax wrote: post #13

I disagree. I'm an observability geek, and OTel is... fine. It's missing a few things that I'd like, but I was able to implement them myself. I guess the major design issue is that the sampling decision is made at the _start_ of the segment. So I hacked up a few improvements: 1. Ability to mark segments as "boring", so they are dropped before the export. For things like healthchecks, empty "get the pending jobs" quer…

OTel has a nice property of working, which sometimes can't be said about vendor stuff.

I think the actual APIs kinda smell at the language level, and when Honeycomb decided to lean into otel and deprecate its Python libs I was super sad, cuz HC's libs were _way_ more usable IMO. Docs are also... painful. Real painful.

I wish that I could get a Python lib which is like "here this is Otel but the config phase isn't weird, and the API just looks a bit better". One of these days.

The biggest trouble I have with Otel recently is getting fixes patched upstream in contrib. Using contribs is super dangeerous, and I would basically recommend people write their own instrumentation and treat the contrib packages as just examples of how to do it

Re: OTel isn’t going well

#116

Post by pphysch »

rtpg wrote: post #114
pphysch wrote: post #100

Earlier quoted context omitted.

There is, it's called VictoriaMetrics/Logs/Traces. https://victoriametrics.com/

We use victoriametrics, but I believe that's just the collector side of it cuz we also query into it with grafana. Datadog isn't just a collector, but the whole querying UI as well, right?

Yes, Grafana seems to be the best free querying frontend across multiple data sources right now. But each Victoria product has its own built-in query GUI as well.

Re: OTel isn’t going well

#117

Post by mnming »

I also think OTel SDKs are a tad bit too prescriptive, but at the same time I can't envision what a better version would look like.

The core of logs and spans are just wide events with some inter-connections, those SDKs and OTel docs make them less obvious.

(I maintain o11ylite https://github.com/o11ylite/o11ylite)

Re: OTel isn’t going well

#118

Post by cyberax »

bilalq wrote: post #93
cyberax wrote: post #85

Earlier quoted context omitted.

I don't use Lambda anymore, but yes. I submitted traces to AWS XRay in a background goroutine with a small timeout.

If you're sending data purely to X-Ray, there's already a daemon running on lambda that you can forward to with low overhead if you don't use OTel. You also get near zero-cost logging and metric to Cloudwatch and EMF. But if you want bring destinations in the mix or do anything other than Cloudwatch , you have to pay the OTel tax. And even if you were content with a pure AWS setup, OTel is still being pushed on you n…

Why do you even _need_ these "layers"? It's a simple RPC protocol that submits data tagged with Span and Trace IDs.

That's really all there is to it. You can just submit it directly, without involving any layers.

Re: OTel isn’t going well

#119

Post by awill88 »

otel is amazing and when thoughtfully instrumented, turns out, you can opt out of auto instrumentation btw, it provides a standard that is useful, well maintained, portable to enterprise or self hosted.

It’s modular, but the author is appraising the Ruby shortcomings as a problem while also saying they unfortunately don’t have time to contribute because, you know, they can’t “join the calls” lol

We just got a CTO who loves Ruby and guess what I’m about to do: use AI to fill in the Ruby gaps and open a PR and work a weekend or two and see if they like it and then you won’t write any more articles disparaging a project that I personally love.

It saves our company AT LEAST 10k a month vs having datadog / splunk / enterprise-y bullcrap. You seem so educated, why not roll up your sleeves instead of patronizing the hard working people that make the project work with your “if it were me” just go ahead and say it in their forums.

And, if you work for your paycheck, you’re using an agent. A mature project like Otel? Shit. That’s easy-peasy to feed into an agent, so what’s what is actually the problem? Take the time to learn and help them out if it bothers you so much you want to share it with the world!

Isn’t every single “problem” found in every large and successful open sourced framework?

Throwing out a baity framing like it’s some kind of project going wrong and then kind of just ending the article without making any sort of judgement on where this all leads, proceeding to post on hackernews.. bait!

It’s a cloud native project that is not owned by any company. That’s so rare and worth an article to celebrate open source! What a privilege to stand on the shoulders of giants!

> So OpenTelemetry currently is attempting to support a dizzying number of languages and frameworks.

“dizzying” — so I’m lost, did the author remmeber the scope of the project before they started making judgements about it?

And calling the attention of hackernews here: what’s the alternative? Oh that’s right, there isn’t one. Because this is a wag my finger article for attention and aggregating the author on a developer channel to boost their presence. Lame.

(Thumbs down)

Re: OTel isn’t going well

#120

Post by suralind »

I don't understand the sentiment. OTEL is better than anything I've ever used before. Do I like every part? No. My personal no-no is the automatic instrumentation which I always bypass and just DI it myself, I don't like the "global" by default approach in Golang and I had to fight team mates who were all for using it. That said, no other observability library that I've ever used was so good overall.

The perf is meh, but tbh if you look at the kind of code we, regular developers write for work, it's probably still vastly better.

Post reply on HN