GitHub CI/CD observability with OpenTelemetry step by step guide • Hacker News Board

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#11

Post by srikanthccv » Sun, Jun 15, 2025, 10:24 AM UTC

sali0 wrote: post #7

noob question, i'm currently adding telemetry to my backend. I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as w…

>I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate

OTeL also has numerous integrations https://opentelemetry.io/ecosystem/registry/. In contrast, Sentry lacks traditional metrics and other capabilities that OTeL offers. IIRC, Sentry experimented with "DDM" (Delightful Developer Metrics), but this feature was deprecated and removed while still in alpha/beta.

Sentry excels at error tracking and provides excellent browser integration. This might be sufficient for your needs, but if you're looking for the comprehensive observability features that OpenTelemetry provides, you'd likely need a full observability platform.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#12

Post by makeavish » Sun, Jun 15, 2025, 10:37 AM UTC

hrpnk wrote: post #4

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

You can use SpanLinks to analyse your async processes. This guide might be helpful introduction: https://dev.to/clericcoder/mastering-trace-analysis-with-spa...

Also SigNoz supports rendering practically unlimited number of spans in trace detail UI and allows filtering them as well which has been really useful in analyzing batch processes: https://signoz.io/blog/traces-without-limits/

You can further run aggregation on spans to monitor failures and latency.

PS: I am SigNoz maintainer

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#13

Post by madduci » Sun, Jun 15, 2025, 10:40 AM UTC

hrpnk wrote: post #4

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

I use Otel running in a GKE cluster and tracking Jenkins jobs, whose spans/traces can track long time running jobs pretty well

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#14

Post by 6r17 » Sun, Jun 15, 2025, 10:41 AM UTC

darkstar_16 wrote: post #6

6r17 wrote: post #5

Earlier quoted context omitted.

I did have some bad experiences with OTEL and have lot of freedom on deployment ; I never read of Signoz will definitely check it out ; SigNoz is working with OTEL I suppose ? I wonder if there are any other adapters for trace injest instead of OTEL ?

Jaeger collector perhaps but then you'd have to use the Jaeger UI. Signoz has a much nicer UI that feels more integrated but last I checked had annoying bugs in the UI like not keeping the time selection when I navigated between screens.

Definitely should look up the tech more ; i lazily commented as Signoz clearly state it ingest most than 50 different sources ;

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#15

Post by elza_1111 » Sun, Jun 15, 2025, 10:56 AM UTC

reactordev wrote: post #2

As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch. That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your o…

There are integrations that let you monitor your AWS resources also on SigNoz. That said, I personally think CloudWatch is painful in so many other ways as well,

Check this out, https://signoz.io/blog/6-silent-traps-inside-cloudWatch-that...

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#16

Post by makeavish » Sun, Jun 15, 2025, 10:57 AM UTC

FunnyLookinHat wrote: post #10

reactordev wrote: post #2

As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch. That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your o…

I think you're looking at OTel from a strictly infrastructure perspective - which Cloudwatch does effectively solve without any added effort. But OTel really begins to shine when you instrument your backends. Some languages (Node.js) have a whole slew of auto-instrumentation, giving you rich traces with spans detailing each step of the http request, every SQL query, and even usage of AWS services. Making those traces…

Agree with you on this. OTel agents allows exporting all host/k8s metrics correlated with your logs and traces. Though exporting AWS service specific metrics with OTel is not easy. To solve this SigNoz has 1-Click AWS Integrations: https://signoz.io/blog/native-aws-integrations-with-autodisc...

Also SigNoz has native correlation between different signals out of the box.

PS: I am SigNoz Maintainer

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#17

Post by elza_1111 » Sun, Jun 15, 2025, 11:10 AM UTC

FunnyLookinHat wrote: post #10

reactordev wrote: post #2

As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch. That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your o…

I think you're looking at OTel from a strictly infrastructure perspective - which Cloudwatch does effectively solve without any added effort. But OTel really begins to shine when you instrument your backends. Some languages (Node.js) have a whole slew of auto-instrumentation, giving you rich traces with spans detailing each step of the http request, every SQL query, and even usage of AWS services. Making those traces…

FYI for anyone reading, OTel does have great auto-instrumentation for Python, Java and .NET also

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#18

Post by elza_1111 » Sun, Jun 15, 2025, 11:22 AM UTC

6r17 wrote: post #5

reactordev wrote: post #2

As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch. That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your o…

I did have some bad experiences with OTEL and have lot of freedom on deployment ; I never read of Signoz will definitely check it out ; SigNoz is working with OTEL I suppose ? I wonder if there are any other adapters for trace injest instead of OTEL ?

yep, SigNoz is OpenTelemetry native. You can instrument your application with OpenTelemetry and send telemetry data direclty to signoz.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#19

Post by dboreham » Sun, Jun 15, 2025, 12:30 PM UTC

sali0 wrote: post #7

noob question, i'm currently adding telemetry to my backend. I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as w…

You can run your own sentry server (or at least last time I worked with it you could). But as others have noted sentry is not going to provide the same functionality as OTel.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#20

Post by totetsu » Sun, Jun 15, 2025, 12:34 PM UTC

I spent some time working on this. First I tried to make a GitHub action that was triggered on completion of your other actions and passed along the context of the triggering action in the environment, then used the GitHub api to call out extra details of the steps and tasks etc, and the logs and make that all into a process trace and send it via an otel connection to like jaeger or grafana, to get flamchart views of performance of steps. I thought maybe it would be better to do this directly from the runner hosts by watching log files, but the api has more detailed information.