41–47 of 47 posts
maybe the first time i read a crystal clear difference between metrics, logs and traces.
nice post.
Has anyone used OpenTelemetry for long-running batch jobs? OTel seems designed for web apps where spans last seconds/minutes, but batch jobs run for hours or days. Since spans are only submitted after completion, there's no way to track progress during execution, making OTel nearly unusable for batch workloads. I have a similar issue with Prometheus -- not great for batch job metrics either. It's frustrating how many…
I've been tasked with adding telemetry to an AWS based service at work: CLI -> Web API Gateway -> Lambda returning a signed S3 URL S3 upload -> SQS -> Lambda which writes to S3 and updates a Dynamo record -> CLI polls for changes This flow isn't only over HTTP and relies on AWS to fire events. I worked around this by embedding the trace ID into the signed URL metadata. It doesn't look like this is possible with all A…
Earlier quoted context omitted.
That's why you sample just enough instead of storing everything
That sounds great until you have a massive issue that costs the company real money and leadership asks why you weren't logging everything in full fidelity? We run with Debug logging on in prod for that reason too. We also ingest insane amounts of data but it does seem to be worth it for a sufficiently complex and important enough system to really have it all.
Has anyone used OpenTelemetry for long-running batch jobs? OTel seems designed for web apps where spans last seconds/minutes, but batch jobs run for hours or days. Since spans are only submitted after completion, there's no way to track progress during execution, making OTel nearly unusable for batch workloads. I have a similar issue with Prometheus -- not great for batch job metrics either. It's frustrating how many…
I think this does highlight, to me, the biggest weakness of OTel--the actual documentation and examples for "how to solve problems with this" really suck.
Earlier quoted context omitted.
True. I think I’m emphasizing their similarities because what I’m seeing is companies treating them as unrelated (eg splunk and signalfx making entirely different query languages and visualization tools for logs vs spans) Imo spans and logs should be understood as the same and displayed and queried the same (it’s trivial to add span id to each log), it almost feels like people are trying to make something trivially s…
Traces and spans can be extended from or added to existing logging, but they aren't the same. Logs are point in time, spans are a duration. Logs are flat, spans have a hierarchy. It's the difference between logging a message in a function, and logging the beginning and end of a function while noting the specific instance of the fn caller. If you have many threads or callers to the same function that difference is cri…
In short, logs can do everything spans can do, depending on how you use them. So really there isn't much distinction.
I'd say that spans and logs are about 95% similar, whereas metrics are wildly dissimilar to both.
Most tools would be be better if they treated spans and logs as two nearly-identical things:
- logs should be viewable in a hierarchy if they have a span id associated - spans should be queryable and countable and alarmable and dashboardable with the same tools as logs
Has anyone used OpenTelemetry for long-running batch jobs? OTel seems designed for web apps where spans last seconds/minutes, but batch jobs run for hours or days. Since spans are only submitted after completion, there's no way to track progress during execution, making OTel nearly unusable for batch workloads. I have a similar issue with Prometheus -- not great for batch job metrics either. It's frustrating how many…
How do you mean? The metrics are available for 15 days by default. What exactly are you missing?