Trying to use OTel in any scenario outside of web backends such as desktop is a frustrating exercise in to trying to find exactly what small subset should use. I wish they had more examples of other types of software.
What Are Traces and Spans in OpenTelemetry?
21–30 of 47 posts
Re: What Are Traces and Spans in OpenTelemetry?
#22Earlier quoted context omitted.
That's why you sample just enough instead of storing everything
That sounds great until you have a massive issue that costs the company real money and leadership asks why you weren't logging everything in full fidelity? We run with Debug logging on in prod for that reason too. We also ingest insane amounts of data but it does seem to be worth it for a sufficiently complex and important enough system to really have it all.
> and leadership asks why you weren't logging everything in full fidelity?
I haven't been asked this question ever. In a way, I wish I was. I wish leadership was engaged in the details of the capabilities of the systems they lead.
But I don't anyone asking me this question any time soon either.
Re: What Are Traces and Spans in OpenTelemetry?
#23This is sort of all just a reframing of existing technologies. Span = an event (which is bascially just a log with an associated trace), and some data fields. Trace = a log for a request with a unique Id. A useful thing about opentelemetry is that there's auto-instrumentation so you can get this all out-of-the-box for most JVM apps. Of course you could probably log your queries instead, so it's not necessarily a game…
I always preach the isomorphism between traces and logs, but you left out the key thing. A span is a log entry associated with a trace, but the other key attributes of the span are its own unique identifier and a reference to the other event that caused the event. With those three attributes you can interpret the trace as a casual graph.
True. I think I’m emphasizing their similarities because what I’m seeing is companies treating them as unrelated (eg splunk and signalfx making entirely different query languages and visualization tools for logs vs spans)
Imo spans and logs should be understood as the same and displayed and queried the same (it’s trivial to add span id to each log), it almost feels like people are trying to make something trivially simple seem more substantial or complex
Re: What Are Traces and Spans in OpenTelemetry?
#24Trying to use OTel in any scenario outside of web backends such as desktop is a frustrating exercise in to trying to find exactly what small subset should use. I wish they had more examples of other types of software.
I agree. An anecdote:
A while ago I was working on some CUDA kernels for n-body physics simulations. It wasn’t too complicated and the end result was generative art. The problem was that it was quite slow and I didn’t know why. Well the core of the application was written in Clojure so I wrote a simple macro to wrap every function in a ns with a span and then ship all the data to jaeger. This ended up being exactly what I needed - I found out that the two slowest functions were data transfer between the GPU memory and writing out a frame (image) to my disk.
In many other places I see the usefulness of this approach but OTel is too often too geared towards HTTP services. Even simple async/queue processing is not as simple. Though, there have been improvements (like span links and trace links).
Re: What Are Traces and Spans in OpenTelemetry?
#25Earlier quoted context omitted.
I always preach the isomorphism between traces and logs, but you left out the key thing. A span is a log entry associated with a trace, but the other key attributes of the span are its own unique identifier and a reference to the other event that caused the event. With those three attributes you can interpret the trace as a casual graph.
True. I think I’m emphasizing their similarities because what I’m seeing is companies treating them as unrelated (eg splunk and signalfx making entirely different query languages and visualization tools for logs vs spans) Imo spans and logs should be understood as the same and displayed and queried the same (it’s trivial to add span id to each log), it almost feels like people are trying to make something trivially s…
Traces and spans can be extended from or added to existing logging, but they aren't the same.
Logs are point in time, spans are a duration. Logs are flat, spans have a hierarchy.
It's the difference between logging a message in a function, and logging the beginning and end of a function while noting the specific instance of the fn caller.
If you have many threads or callers to the same function that difference is critical in tracing causality of failures or any other type of action of note.
Re: What Are Traces and Spans in OpenTelemetry?
#26The amount of additional code that it needs is horrible. We will now have to spend more brain juice on telemetry when working on a feature.
There’s certainly some overhead, nothing is free. But the tradeoff is better insight into your system and better tools to validate issues when they arise. It can be very powerful in those scenarios.
Ive spent countless hours on issues where customers complain about performance or a bug and it just can’t be reproduced. Telemetry allows us to get more information to locate and fix these issues.
Re: What Are Traces and Spans in OpenTelemetry?
#27Is there anything that wraps multiple requests?
Re: What Are Traces and Spans in OpenTelemetry?
#28The amount of additional code that it needs is horrible. We will now have to spend more brain juice on telemetry when working on a feature.
Thanks for your comment! It has given me an idea for a project: a simple library that provides a Python decorator that can be used to include basic telemetry for functions in Python code (think spans with the input parameters as attributes): https://github.com/diegojromerolopez/otelize
Feedback welcome!
Re: What Are Traces and Spans in OpenTelemetry?
#29This is sort of all just a reframing of existing technologies. Span = an event (which is bascially just a log with an associated trace), and some data fields. Trace = a log for a request with a unique Id. A useful thing about opentelemetry is that there's auto-instrumentation so you can get this all out-of-the-box for most JVM apps. Of course you could probably log your queries instead, so it's not necessarily a game…
Span has a beginning and an end time.
Event typically just has a time when it happened.
Re: What Are Traces and Spans in OpenTelemetry?
#30Earlier quoted context omitted.
That's why you sample just enough instead of storing everything
That sounds great until you have a massive issue that costs the company real money and leadership asks why you weren't logging everything in full fidelity? We run with Debug logging on in prod for that reason too. We also ingest insane amounts of data but it does seem to be worth it for a sufficiently complex and important enough system to really have it all.
> That sounds great until you have a massive issue that costs the company real money and leadership asks why you weren't logging everything in full fidelity?
You should have an answer, right? Like, in your case, you run a lot of logging, and you know why. So if it's off, you say "because it would cost X/million dollars a year and we decided not to do it."
Course, if you're the one who set it up, you should have the receipts on when that decision was made. This can be tricky sometimes because a lot of software dev ICs are strangely insulated from direct budgets, but if you're presented with an option that would be helpful but would cost a ton of money, it's generally a good thing to at least quickly run by someone higher up to confirm the desired direction.