Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Are Traces and Spans in OpenTelemetry?

oneuptime.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: What Are Traces and Spans in OpenTelemetry?

#31

Post by drivenextfunc »

Has anyone used OpenTelemetry for long-running batch jobs? OTel seems designed for web apps where spans last seconds/minutes, but batch jobs run for hours or days. Since spans are only submitted after completion, there's no way to track progress during execution, making OTel nearly unusable for batch workloads.

I have a similar issue with Prometheus -- not great for batch job metrics either. It's frustrating how many otherwise excellent OSS tools are optimized for web applications but fall short for batch processing use cases.

Re: What Are Traces and Spans in OpenTelemetry?

#32

Post by no_wizard »

evidencetamper wrote: post #22
voidfunc wrote: post #16

Earlier quoted context omitted.

That sounds great until you have a massive issue that costs the company real money and leadership asks why you weren't logging everything in full fidelity? We run with Debug logging on in prod for that reason too. We also ingest insane amounts of data but it does seem to be worth it for a sufficiently complex and important enough system to really have it all.

> and leadership asks why you weren't logging everything in full fidelity? I haven't been asked this question ever. In a way, I wish I was. I wish leadership was engaged in the details of the capabilities of the systems they lead. But I don't anyone asking me this question any time soon either.

Have you ever been asked “why didn’t we catch this sooner?”. I feel like it’s the same question worded differently

Re: What Are Traces and Spans in OpenTelemetry?

#34

Post by mdaniel »

andoando wrote: post #27

Is there anything that wraps multiple requests?

I doubt "wraps" but almost certainly what you're shopping for is a correlation identifier on the (logs, traces, metrics) that would enable you to group the related requests. Sometimes just the session id can get you where you want to go, but in more complicated setups you may have to annotate from the client side to indicate "I'm doing these 5 things as part of this one logical operation"

Re: What Are Traces and Spans in OpenTelemetry?

#35

Post by nickzelei »

drivenextfunc wrote: post #31

Has anyone used OpenTelemetry for long-running batch jobs? OTel seems designed for web apps where spans last seconds/minutes, but batch jobs run for hours or days. Since spans are only submitted after completion, there's no way to track progress during execution, making OTel nearly unusable for batch workloads. I have a similar issue with Prometheus -- not great for batch job metrics either. It's frustrating how many…

Hm from what I’ve seen it emits metrics at a regular interval just like Prometheus. Maybe I’m thinking of something else though.

Re: What Are Traces and Spans in OpenTelemetry?

#36

Post by ekzy »

drivenextfunc wrote: post #31

Has anyone used OpenTelemetry for long-running batch jobs? OTel seems designed for web apps where spans last seconds/minutes, but batch jobs run for hours or days. Since spans are only submitted after completion, there's no way to track progress during execution, making OTel nearly unusable for batch workloads. I have a similar issue with Prometheus -- not great for batch job metrics either. It's frustrating how many…

I’ve implemented OTEL for background jobs, so async jobs that get picked up from the DB where I store the trace context in the DB and pass it along to multiple async jobs. For some jobs that fail and retry with a backoff strategy, they can take many hours and we can see the traces fine in grafana. Each job create its own span but they are all within the same trace.

Works well for us, I’m not sure I understand the issue you’re facing?

Re: What Are Traces and Spans in OpenTelemetry?

#37

Post by ekzy »

ekzy wrote: post #36
drivenextfunc wrote: post #31

Has anyone used OpenTelemetry for long-running batch jobs? OTel seems designed for web apps where spans last seconds/minutes, but batch jobs run for hours or days. Since spans are only submitted after completion, there's no way to track progress during execution, making OTel nearly unusable for batch workloads. I have a similar issue with Prometheus -- not great for batch job metrics either. It's frustrating how many…

I’ve implemented OTEL for background jobs, so async jobs that get picked up from the DB where I store the trace context in the DB and pass it along to multiple async jobs. For some jobs that fail and retry with a backoff strategy, they can take many hours and we can see the traces fine in grafana. Each job create its own span but they are all within the same trace. Works well for us, I’m not sure I understand the iss…

Ok after re reading I think you have issues with long running spans, I think you should break down your spans in smaller chunks. But a trace can take many hours or days, and be analysed even when it’s not finished

Re: What Are Traces and Spans in OpenTelemetry?

#38

Post by voidfunc »

no_wizard wrote: post #32
evidencetamper wrote: post #22

Earlier quoted context omitted.

> and leadership asks why you weren't logging everything in full fidelity? I haven't been asked this question ever. In a way, I wish I was. I wish leadership was engaged in the details of the capabilities of the systems they lead. But I don't anyone asking me this question any time soon either.

Have you ever been asked “why didn’t we catch this sooner?”. I feel like it’s the same question worded differently

Its really two questions:

1. Why didn't we catch this sooner

2. Why did it take so long to mitigate

Without the debug logging #2 can be really tricky sometimes as well as you can be flying blind to some deep internal conditional branch firing off.

Re: What Are Traces and Spans in OpenTelemetry?

#39

Post by digianarchist »

I've been tasked with adding telemetry to an AWS based service at work:

CLI -> Web API Gateway -> Lambda returning a signed S3 URL S3 upload -> SQS -> Lambda which writes to S3 and updates a Dynamo record -> CLI polls for changes

This flow isn't only over HTTP and relies on AWS to fire events. I worked around this by embedding the trace ID into the signed URL metadata. It doesn't look like this is possible with all AWS services.

I wonder if X-Ray can help here?

It can also be tedious to initialize spans everywhere. Aspects could help a lot here and orchestrion [0] is a good example of how it could be done in Go. I haven't found an OTEL equivalent yet (though haven't looked hard).

[0] - https://datadoghq.dev/orchestrion/docs/architecture/#code-in...

Re: What Are Traces and Spans in OpenTelemetry?

#40

Post by freetonik »

tnolet wrote: post #20
zug_zug wrote: post #9

This is sort of all just a reframing of existing technologies. Span = an event (which is bascially just a log with an associated trace), and some data fields. Trace = a log for a request with a unique Id. A useful thing about opentelemetry is that there's auto-instrumentation so you can get this all out-of-the-box for most JVM apps. Of course you could probably log your queries instead, so it's not necessarily a game…

yeah, but spans can have events!

So, events are recursive?
Post reply on HN