GitHub CI/CD observability with OpenTelemetry step by step guide
1–10 of 60 posts
Re: GitHub CI/CD observability with OpenTelemetry step by step guide
#2As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch.
That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your org, it’s going to be extremely painful to switch. Choose otel if you’re a hybrid cloud or you have on premises stuff. If you’re on AWS, CloudWatch is a better option simply because they have the data. Dead simple tracing.
Re: GitHub CI/CD observability with OpenTelemetry step by step guide
#3That's a genius idea. So obvious in retrospect.
Re: GitHub CI/CD observability with OpenTelemetry step by step guide
#4Has anyone seen OTel being used well for long-running batch/async processes?
Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.
Re: GitHub CI/CD observability with OpenTelemetry step by step guide
#5As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch. That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your o…
I did have some bad experiences with OTEL and have lot of freedom on deployment ; I never read of Signoz will definitely check it out ; SigNoz is working with OTEL I suppose ?
I wonder if there are any other adapters for trace injest instead of OTEL ?
Re: GitHub CI/CD observability with OpenTelemetry step by step guide
#6As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch. That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your o…
I did have some bad experiences with OTEL and have lot of freedom on deployment ; I never read of Signoz will definitely check it out ; SigNoz is working with OTEL I suppose ? I wonder if there are any other adapters for trace injest instead of OTEL ?
Jaeger collector perhaps but then you'd have to use the Jaeger UI. Signoz has a much nicer UI that feels more integrated but last I checked had annoying bugs in the UI like not keeping the time selection when I navigated between screens.
Re: GitHub CI/CD observability with OpenTelemetry step by step guide
#7noob question, i'm currently adding telemetry to my backend.
I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as well.
Curious if anyone has thoughts on this. Assuming Sentry can fulfill our requirements, the only thing taht really concerns me is vendor-lockin. But I'm wondering other people's thoughts
Re: GitHub CI/CD observability with OpenTelemetry step by step guide
#8Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.
I've tried and failed at tracing transactions that span multiple queues (with different backends). At the end I just published some custom metrics for the transaction's success count / failure count / duration and moved on my with life.
Re: GitHub CI/CD observability with OpenTelemetry step by step guide
#9noob question, i'm currently adding telemetry to my backend. I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as w…
Sentry isn’t really a full on observability platform. It’s for error reporting only (that is annotated with traces and logs). It turns out that for most projects, this is sufficient. Can’t comment on the vendor lock-in part.
Re: GitHub CI/CD observability with OpenTelemetry step by step guide
#10As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch. That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your o…
I think you're looking at OTel from a strictly infrastructure perspective - which Cloudwatch does effectively solve without any added effort. But OTel really begins to shine when you instrument your backends. Some languages (Node.js) have a whole slew of auto-instrumentation, giving you rich traces with spans detailing each step of the http request, every SQL query, and even usage of AWS services. Making those traces even more valuable is that they're linked across services.
We've frequently seen a slowdown or error at the top of our stack, and the teams are able to immediately pinpoint the problem as a downstream service. Not only that, they can see the specific issue in the downstream service almost immediately!
Once you get to that level of detail, having your infrastructure metrics pulled into your Otel provider does start to make some sense. If you observe a slowdown in a service, being able to see that the DB CPU is pegged at the same time is meaningful, etc.
[Edit - Typo!]