Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub CI/CD observability with OpenTelemetry step by step guide

signoz.io

Post reply on HN
Print view View original

51–60 of 60 posts

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#51

Post by remram »

sweetgiorni wrote: post #50
remram wrote: post #30

I have thought about that before, but I was blocked by the really poor file support for OTel. I couldn't find an easy way to dump a file from the collector running in my CI job and load it on my laptop for analysis, which is the way I would like to go. Maybe this has changed?

https://github.com/open-telemetry/opentelemetry-collector-co...

And the receiver: https://github.com/open-telemetry/opentelemetry-collector-co...

I'll have to try this!

edit: actually Jaeger can just read those files directly, so no need to run a collector with the receiver. This is great!

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#52

Post by stackskipton »

avtar wrote: post #44
stackskipton wrote: post #37

Earlier quoted context omitted.

Ops type here, Otel is great but if your metrics are not there, please fix that. In particular, consider just import prometheus_client and going from there. Prometheus is bog easy to run, Grafana understands it and anything involving alerting/monitoring from logs is bad idea for future you, I PROMISE YOU, PLEASE DON'T!

> anything involving alerting/monitoring from logs is bad idea for future you Why is issuing alerts for log events a bad idea?

Couple of reasons.

Biggest one, sample rate is much higher (every log) and this can cause problems if service goes haywire and starts spewing logs everywhere. Logging pipelines tend to be very rigid as well for various reasons. Metrics are easier to handle as you can step back sample rate, drop certain metrics or spin up additional Prometheus instances.

Logging format becomes very rigid and if the company goes multiple languages, this can be problematic as different languages can behave differently. Is this exception something we care about or not? So we throw more code in attempt to get logging alerting into state that does not drive everyone crazy where if we were just doing "rate(critical_errors[5m] > 10" in Prometheus, we would be all set!

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#53

Post by brunoqc »

makeavish wrote: post #12
hrpnk wrote: post #4

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

You can use SpanLinks to analyse your async processes. This guide might be helpful introduction: https://dev.to/clericcoder/mastering-trace-analysis-with-spa... Also SigNoz supports rendering practically unlimited number of spans in trace detail UI and allows filtering them as well which has been really useful in analyzing batch processes: https://signoz.io/blog/traces-without-limits/ You can further run aggregation…

Is SpanLinks supported by jaeger?

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#54

Post by makeavish »

brunoqc wrote: post #53
makeavish wrote: post #12

Earlier quoted context omitted.

You can use SpanLinks to analyse your async processes. This guide might be helpful introduction: https://dev.to/clericcoder/mastering-trace-analysis-with-spa... Also SigNoz supports rendering practically unlimited number of spans in trace detail UI and allows filtering them as well which has been really useful in analyzing batch processes: https://signoz.io/blog/traces-without-limits/ You can further run aggregation…

Is SpanLinks supported by jaeger?

Yes Jaeger supports SpanLinks

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#55

Post by sali0 »

stackskipton wrote: post #37
sali0 wrote: post #7

noob question, i'm currently adding telemetry to my backend. I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as w…

Ops type here, Otel is great but if your metrics are not there, please fix that. In particular, consider just import prometheus_client and going from there. Prometheus is bog easy to run, Grafana understands it and anything involving alerting/monitoring from logs is bad idea for future you, I PROMISE YOU, PLEASE DON'T!

Thank you, this is where I'll likely start.

From other comments as well, seems it's still worth trying to integrate otel. Appreciate everyone's insights

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#57

Post by vanschelven »

sali0 wrote: post #7

noob question, i'm currently adding telemetry to my backend. I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as w…

I'd say "track errors first" [0] and focus on APM later (if at all). If you're worried about Sentry's lock-in, know that there are API-compatible drop-in replacements[1][2] though they are less feature-complete on the APM/observability side.

[0] https://www.bugsink.com/blog/track-errors-first/

[1] https://www.bugsink.com/

[2] https://glitchtip.com/

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#58

Post by atypeoferror »

hrpnk wrote: post #4

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

We have, and I won’t lie, it’s kinda a pain. The reality we discovered is that async flows are second-class citizens in otel tracing.

We essentially propagate the context manually between APIs and lambdas, through HTTP headers, SNS/SQS and even storage. We also ended up pulling Tempo’s Parquet files (we’re self-hosting the grafana stack) into Redash to be able to do real analysis. We got a lot of great insights and were able to do a lot of tuning that would have been impossible otherwise, but it was quite an investment. Would love to know if there is anything out there that would have made this less painful.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#60

Post by drzaiusx11 »

hrpnk wrote: post #4

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

yes, the only tricky part for us were the short async jobs where the otel queue didn’t have time to process its events before the tasks exited. we just added a drain wait loop in our at_exit callbacks
Post reply on HN