Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub CI/CD observability with OpenTelemetry step by step guide

signoz.io

Post reply on HN
Print view View original

31–40 of 60 posts

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#31

Post by mdaniel »

bbkane wrote: post #26
6r17 wrote: post #5

Earlier quoted context omitted.

I did have some bad experiences with OTEL and have lot of freedom on deployment ; I never read of Signoz will definitely check it out ; SigNoz is working with OTEL I suppose ? I wonder if there are any other adapters for trace injest instead of OTEL ?

There are a few: I've played with https://uptrace.dev and https://openobserve.ai/ . OpenObserve is a single binary, so easy to set up

be cognizant of their licenses (AGPLv3), it matters in some shops

https://github.com/uptrace/uptrace/blob/v1.7.6/LICENSE

https://github.com/openobserve/openobserve/blob/v0.14.7/LICE...

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#32

Post by mdaniel »

reactordev wrote: post #2

As someone who has some experience in observability at scale, the issue with SigNoz, Prom, etc is that they can only operate on the data that is exposed by the underlying infrastructure where the IaaS has all the information to provide a better experience. Hence CloudWatch. That said, if you own your infrastructure, I’d build out a signoz cluster in a heartbeat. Otel is awesome but once you set down a path for your o…

A child comment mentioned k8s but I also have been chomping at the bit to try out the eBPF hooks in https://github.com/pixie-io/pixie (or even https://github.com/coroot/coroot or https://github.com/parca-dev/parca ) all of which are Apache 2 licensed

The demo for https://github.com/draios/sysdig was also just amazing, but I don't have any idea what the storage requirements would be for leaving it running

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#33

Post by mdaniel »

dboreham wrote: post #19
sali0 wrote: post #7

noob question, i'm currently adding telemetry to my backend. I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as w…

You can run your own sentry server (or at least last time I worked with it you could). But as others have noted sentry is not going to provide the same functionality as OTel.

The word "can" is doing a lot of work in your comment, based on the now horrific number of moving parts[1] and I think David has even said the self-hosting story isn't a priority for them. Also, don't overlook the license, if your shop is sensitive to non-FOSS licensing terms

1: https://github.com/getsentry/self-hosted/blob/25.5.1/docker-...

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#34

Post by hinkley »

dboreham wrote: post #21
hrpnk wrote: post #4

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

It doesn't matter how long things take. The best way to understand this is to realize that OTel tracing (and all other similar things) are really "fancy logging systems". Some agent code emits a log message every time something happens (e.g. batch job begins, batch job ends). Something aggregates those log messages into some place they can be coherently scanned. Then something scans those messages generating some vis…

Every time people talk about OTel I discover half the people are talking about spans rather that stats. For stats it’s not a ‘fancy logger’ because it’s condensing the data at various steps.

And if you’ve ever tried to trace a call tree using correlationIDs and Splunk queries and still say OTEL is ‘just a fancy’ then you’re in dangerous territory, even if it’s just by way of explanation. Don’t feed the masochists. When masochists derail attempts at pain reduction they become sadists.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#35

Post by BoiledCabbage »

candiddevmike wrote: post #24
imiric wrote: post #22

Earlier quoted context omitted.

That's a good way of looking at it, but it assumes that both start and end events will be emitted and will successfully reach the backend. What happens if one of them doesn't?

AIUI, there aren't really start or end messages, they're spans. A span is technically an "end" message and will have parent or child spans.

I don't know the details but does a span have a beginning?

Is that beginning "logged" at a separate point in time from when the span end is logged?

> AIUI, there aren't really start or end messages,

Can you explain this sentence a bit more? How does it have a duration without a start and end?

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#36

Post by hinkley »

imiric wrote: post #22
dboreham wrote: post #21

Earlier quoted context omitted.

It doesn't matter how long things take. The best way to understand this is to realize that OTel tracing (and all other similar things) are really "fancy logging systems". Some agent code emits a log message every time something happens (e.g. batch job begins, batch job ends). Something aggregates those log messages into some place they can be coherently scanned. Then something scans those messages generating some vis…

That's a good way of looking at it, but it assumes that both start and end events will be emitted and will successfully reach the backend. What happens if one of them doesn't?

Ugh. One of the reasons I never turned on the tracing code I painstakingly refactored into our stats code was discovering that OTEL makes no attempts to introduce a span to the collector prior to child calls talking about it. Is that really how you want to do event correlation? Time traveling seems like an expensive operation when you’re dealing with 50,000 trace events per second.

The other turns out to be our OPs teams problem more than OTEL’s. Well a little of both. If a trace goes over a limit then OTEL just silently drops the entire thing, and the default size on AWS is useful for toy problems not retrofitting onto live systems. It’s the silent failure defaults of OTEL that are giant footguns. Give me a fucking error log on data destruction, you asshats.

I’ll just use Prometheus next time, which is apparently what our OPs team recommended (except one individual who was the one I talked to).

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#37

Post by stackskipton »

sali0 wrote: post #7

noob question, i'm currently adding telemetry to my backend. I was at first implementing otel throughout my api, but ran into some minor headaches and a lot of boilerplate. I shopped a bit around and saw that Sentry has a lot of nice integrations everywhere, and seems to have all the same features (metrics, traces, error reporting). I'm considering just using Sentry for both backend and frontend and other pieces as w…

Ops type here, Otel is great but if your metrics are not there, please fix that. In particular, consider just import prometheus_client and going from there.

Prometheus is bog easy to run, Grafana understands it and anything involving alerting/monitoring from logs is bad idea for future you, I PROMISE YOU, PLEASE DON'T!

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#38

Post by hinkley »

BoiledCabbage wrote: post #35
candiddevmike wrote: post #24

Earlier quoted context omitted.

AIUI, there aren't really start or end messages, they're spans. A span is technically an "end" message and will have parent or child spans.

I don't know the details but does a span have a beginning? Is that beginning "logged" at a separate point in time from when the span end is logged? > AIUI, there aren't really start or end messages, Can you explain this sentence a bit more? How does it have a duration without a start and end?

It’s been a minute since I worked on this but IIRC no, which means that if the request times out you have to be careful to end the span, and also all of the dependent calls show up at the collector in reverse chronological order.

The thing is that at scale you’d never be able to guarantee that the start of the span showed up at a collector in chronological order anyway, especially due to the queuing intervals being distinct per collection sidecar. But what you could do with two events is discover spans with no orderly ending to them. You could easily truncate traces that go over the span limit instead of just dropping them on the floor (fuck you for this, OTEL, this is the biggest bullshit in the entire spec). And you could reduce the number of traceids in your parsing buffer that have no metadata associated with them, both in aggregate and number of messages in the limbo state per thousand events processed.

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#39

Post by ai-christianson »

makeavish wrote: post #12
hrpnk wrote: post #4

Has anyone seen OTel being used well for long-running batch/async processes? Wonder how the suggestions stack up to monolith builds for Apps that take about an hour.

You can use SpanLinks to analyse your async processes. This guide might be helpful introduction: https://dev.to/clericcoder/mastering-trace-analysis-with-spa... Also SigNoz supports rendering practically unlimited number of spans in trace detail UI and allows filtering them as well which has been really useful in analyzing batch processes: https://signoz.io/blog/traces-without-limits/ You can further run aggregation…

Is this better than Honeycomb?

Re: GitHub CI/CD observability with OpenTelemetry step by step guide

#40

Post by nijave »

hinkley wrote: post #36
imiric wrote: post #22

Earlier quoted context omitted.

That's a good way of looking at it, but it assumes that both start and end events will be emitted and will successfully reach the backend. What happens if one of them doesn't?

Ugh. One of the reasons I never turned on the tracing code I painstakingly refactored into our stats code was discovering that OTEL makes no attempts to introduce a span to the collector prior to child calls talking about it. Is that really how you want to do event correlation? Time traveling seems like an expensive operation when you’re dealing with 50,000 trace events per second. The other turns out to be our OPs t…

You can usually turn logging on but a lot of the OTEL stack defaults to best effort and silently drops data.

We had Grafana Agent running which was wrapping the reference implementation OTEL collector written in go and it was pretty easy to see when data was being dropped via logs.

I think some limitation is also on the storage backend. We were using Grafana Cloud Tempo which imposes limits. I'd think using a backend that doesn't enforce recency would help.

With the OTEL collector I'd think you could utilize some processors/connectors or write your own to handle individual spans that get too big. Not sure on backends but my current company uses Datadog and their proprietary solution handles >30k spans per trace pretty easily.

I think the biggest issue is the low cohesion, high DIY nature of OTEL. You can build powerful solutions but you really need to get low level and assemble everything yourself tuning timeouts, limits, etc for your use case.

Post reply on HN