Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub CI/CD observability with OpenTelemetry step by step guide

signoz.io

Post reply on HN
Print view View original

41–50 of 60 posts

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#41

Post by nijave »

BoiledCabbage wrote: post #35
candiddevmike wrote: post #24

Earlier quoted context omitted.

AIUI, there aren't really start or end messages, they're spans. A span is technically an "end" message and will have parent or child spans.

I don't know the details but does a span have a beginning? Is that beginning "logged" at a separate point in time from when the span end is logged? > AIUI, there aren't really start or end messages, Can you explain this sentence a bit more? How does it have a duration without a start and end?

A span is a discrete event emitted on completion. It contains arbitrary metadata (plus a few mandatory fields if you're following the OTEL spec).

As such, it doesn't really have a beginning or end except that it has fields for duration and timestamps.

I'd check out the OTEL docs since I think seeing the examples as JSON helps clarify things. It looks like they have events attached to spans which is optional. https://opentelemetry.io/docs/concepts/signals/traces/

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#42

Post by hinkley »

nijave wrote: post #40
hinkley wrote: post #36

Earlier quoted context omitted.

Ugh. One of the reasons I never turned on the tracing code I painstakingly refactored into our stats code was discovering that OTEL makes no attempts to introduce a span to the collector prior to child calls talking about it. Is that really how you want to do event correlation? Time traveling seems like an expensive operation when you’re dealing with 50,000 trace events per second. The other turns out to be our OPs t…

You can usually turn logging on but a lot of the OTEL stack defaults to best effort and silently drops data. We had Grafana Agent running which was wrapping the reference implementation OTEL collector written in go and it was pretty easy to see when data was being dropped via logs. I think some limitation is also on the storage backend. We were using Grafana Cloud Tempo which imposes limits. I'd think using a backend…

> I think the biggest issue is the low cohesion, high DIY nature of OTEL

OTEL is the SpringBoot of telemetry and if you think those are fighting words then I picked the right ones.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#43

Post by mdaniel »

ai-christianson wrote: post #39
makeavish wrote: post #12

Earlier quoted context omitted.

You can use SpanLinks to analyse your async processes. This guide might be helpful introduction: https://dev.to/clericcoder/mastering-trace-analysis-with-spa... Also SigNoz supports rendering practically unlimited number of spans in trace detail UI and allows filtering them as well which has been really useful in analyzing batch processes: https://signoz.io/blog/traces-without-limits/ You can further run aggregation…

Is this better than Honeycomb?

"Better" is always "for what metric" but if nothing else having the source code to the stack is always "better" IMHO even if one doesn't choose to self-host, and that goes double for SigNoz choosing a permissive license, so one doesn't have to get lawyers involved to run it

---

While digging into Honeycomb's open source story, I did find these two awesome toys, one relevant to the otel discussion and one just neato

https://github.com/honeycombio/refinery (Apache 2) -- Refinery is a tail-based sampling proxy and operates at the level of an entire trace. Refinery examines whole traces and intelligently applies sampling decisions to each trace. These decisions determine whether to keep or drop the trace data in the sampled data forwarded to Honeycomb.

https://github.com/honeycombio/gritql (MIT) -- GritQL is a declarative query language for searching and modifying source code

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#44

Post by avtar »

stackskipton wrote: post #37
sali0 wrote: post #7

noob question, i'm currently adding telemetry to my backend. I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as w…

Ops type here, Otel is great but if your metrics are not there, please fix that. In particular, consider just import prometheus_client and going from there. Prometheus is bog easy to run, Grafana understands it and anything involving alerting/monitoring from logs is bad idea for future you, I PROMISE YOU, PLEASE DON'T!

> anything involving alerting/monitoring from logs is bad idea for future you

Why is issuing alerts for log events a bad idea?

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#45

Post by _kblcuk_ »

avtar wrote: post #44
stackskipton wrote: post #37

Earlier quoted context omitted.

Ops type here, Otel is great but if your metrics are not there, please fix that. In particular, consider just import prometheus_client and going from there. Prometheus is bog easy to run, Grafana understands it and anything involving alerting/monitoring from logs is bad idea for future you, I PROMISE YOU, PLEASE DON'T!

> anything involving alerting/monitoring from logs is bad idea for future you Why is issuing alerts for log events a bad idea?

It’s trivial to alter or remove log lines without knowing or realizing that it affects some alerting or monitoring somewhere. That’s why there are dedicated monitoring and alerting systems to start with.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#46

Post by sethammons »

hrpnk wrote: post #4

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

We had a hell of a time attempting to roll out OTel for that kind of work. Our scale was also billions of requests per day.

We ended up taking tracing out of these jobs, and only using on requests that finish in short order, like UI web requests. For our longer jobs and fanout work, we started passing a metadata object around that appended timing data related that specific job and then at egress, would capture the timing metadata and flag abnormalities.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#47

Post by reactordev »

FunnyLookinHat wrote: post #10
reactordev wrote: post #2

As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch. That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your o…

I think you're looking at OTel from a strictly infrastructure perspective - which Cloudwatch does effectively solve without any added effort. But OTel really begins to shine when you instrument your backends. Some languages (Node.js) have a whole slew of auto-instrumentation, giving you rich traces with spans detailing each step of the http request, every SQL query, and even usage of AWS services. Making those traces…

Not confusing anything. Yes you can meter your own applications, generate your own metrics, but most organizations start their observability journey with the hardware and latency metrics.

Otel provides a means to sugar any metric with labels and attributes which is great (until you have high cardinality) but there are still things that are at the infrastructure level that only CloudWatch knows of (on AWS). If you’re running K8s on your own hardware - Otel would be my first choice.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#48

Post by sethammons »

_kblcuk_ wrote: post #45
avtar wrote: post #44

Earlier quoted context omitted.

> anything involving alerting/monitoring from logs is bad idea for future you Why is issuing alerts for log events a bad idea?

It’s trivial to alter or remove log lines without knowing or realizing that it affects some alerting or monitoring somewhere. That’s why there are dedicated monitoring and alerting systems to start with.

Same with metrics.

If you need an artifact from your system, it should be tested. We test our logs and many types of metrics. Too many incidents from logs or metrics changing and no longer causing alerts. Never got to build out my alert test bed that exercises all know alerts in prod, verifying they continue to work.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#50

Post by sweetgiorni »

remram wrote: post #30

I have thought about that before, but I was blocked by the really poor file support for OTel. I couldn't find an easy way to dump a file from the collector running in my CI job and load it on my laptop for analysis, which is the way I would like to go. Maybe this has changed?

https://github.com/open-telemetry/opentelemetry-collector-co...
Post reply on HN