This is stuff that a debugger is supposed to do for you, for free. This should not require code at the application level, but it should be implemented at the tooling level.
Are you running a debugger on a web service in production?
Tracing: Structured logging, but better
81–90 of 130 posts
Re: Tracing: Structured logging, but better
#82One thing about logging and tracing is the inevitable cost (in real money). I love observability probably more than most. And my initial reaction to this article is the obvious: why not both? In fact, I tend to think more in terms of "events" when writing both logs and tracing code. How that event is notified, stored, transmitted, etc. is in some ways divorced from the activity. I don't care if it is going to stdout,…
FWIW part of the reason you're seeing that is, at least traditionally, APM companies rebranding as Observability companies stuffed trace data into metrics data stores, which becomes prohibitively expensive to query with custom tags/attributes/fields. Newer tools/companies have a different approach that makes cost far more predictable and generally lower. Luckily, some of the larger incumbents are also moving away fro…
> Newer tools/companies have a different approach that makes cost far more predictable and generally lower.
What newer tools/companies are in this category? Any that you recommend?
Re: Tracing: Structured logging, but better
#83This is stuff that a debugger is supposed to do for you, for free. This should not require code at the application level, but it should be implemented at the tooling level.
Are you running a debugger on a web service in production?
Tracing would be on every web service in production! At the same time! And saving all the output!
Re: Tracing: Structured logging, but better
#84How would a hobbyist programmer get started with tracing for a simple web app? Where do the traces end up and how do I query it? Can tracing be used in a development environment? Context: the last thing I wrote used Deno and Deno Deploy.
Just install opentelemetry libs. I found this example with a quick search: https://dev.to/grunet/leveraging-opentelemetry-in-deno-45bj opentelemetry has a service you can run that will collect the telemetry data and you can export it to something like prometheus which can store it and let you query it. Example here https://github.com/open-telemetry/opentelemetry-collector-co... Typically in dev environments trace spa…
For local dev Jaegar provides a very nice UI to do trace inspection, searching etc.
Re: Tracing: Structured logging, but better
#85Minor nitpick, but I wish this post started with defining what we mean by logging vs tracing, since some people use these interchangeably. The reader instead has to infer this from the criticisms of logging.
I've never encountered this confusion anywhere, so I wouldn't ever think to dispel it. Which isn't to say that I disagree with the more general point that defining your terms is good thing. In any case, the post itself (which is not long) illustrates and marks out many of the differences.
I would guess that you're either not around junior engineers, or people are very good at hiding their confusion.
Re: Tracing: Structured logging, but better
#86Tracing is poor at both very long lived traces, at stream processing, and most tracing implementations are too heavy to run in computationally bound tasks beyond at a very coarse level. Logging is nice in that it has no context, no overhead, is generally very cheap to compose and emit, and with including transaction id and done in a structured way gives you most of what tracing does without all the other baggage. Tha…
Hmmm, for long-lived processes and stream processing we use tracing just fine. What we do is make a cutoff of 60 seconds, which each chunk is its own trace. But our backend queries trace data directly, so we can still analyze the aggregate, long-term behavior and then dig into a particular 60 second chunk if it's problematic.
So, here are a few examples -
Suppose you have a long data pipeline that you want to trace jobs across. There are not an enormous number of jobs but each one takes 12 hours across many phases. In theory tracing works great here, but in practice most tracing platforms can’t handle this. This is especially true with tailed based tracing as traces can be unbounded and it has to assume at some point their time out. You can certainly build your own, but most of the value of tracing solutions is the user experience; which is also the hardest part.
On stream processing I’ve generally found it too expensive to instrument stream processors with tracing. Also there’s generally not enough variability to make it interesting. Context stitching and span management as well as sweeping and shipping of traces can be expensive in a lot of implementations and stream processing is often cpu bound.
A simple transaction id annotated log makes a lot more sense in both, queried in a log analytic platform.
Re: Tracing: Structured logging, but better
#87Earlier quoted context omitted.
FWIW part of the reason you're seeing that is, at least traditionally, APM companies rebranding as Observability companies stuffed trace data into metrics data stores, which becomes prohibitively expensive to query with custom tags/attributes/fields. Newer tools/companies have a different approach that makes cost far more predictable and generally lower. Luckily, some of the larger incumbents are also moving away fro…
> Newer tools/companies have a different approach that makes cost far more predictable and generally lower. What newer tools/companies are in this category? Any that you recommend?
I haven't used anything else, but I'll gladly shill for https://honeycomb.io.
Re: Tracing: Structured logging, but better
#88Logs should go to stderr. I will die on this hill.
Re: Tracing: Structured logging, but better
#89> If you’re writing log statements, you’re doing it wrong. I too use this bait statement. Then I follow it up with (the short version): 1) Rewrite your log statements so that they're machine readable 2) Prove they're machine-readable by having the down-stream services read them instead of the REST call you would have otherwise sent. 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for y…
> 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for you. I don't think this is a reasonable statement. There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka. Bringing up Kafka sounds like a case of a solution looking for a problem.
> I don't think this is a reasonable statement. There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka. Bringing up Kafka sounds like a case of a solution looking for a problem.
If it's data you care about then you put it in Kafka, unless you're big enough to use something like Cassandra or rich enough to pay a cloud provider to make redundant data storage their problem. Logs are something that you need to write durably and reliably when shit is hitting the fan and your networks are flaking and machines are crashing - so ephemeral disks are out, NFS is out, ad-hoc log collector gossip protocols are out, and anything that relies on single master -> read replica and "promoting" that replica is definitely out.
Kafka is about as lightweight as it gets for anything that can't be single-machine/SPOF. It's a lot simpler and more consistent than any RDBMS. What else would you use? HDFS (or maybe OpenAFS if your ops team is really good) is the only half-reasonable alternative I can think of.
Re: Tracing: Structured logging, but better
#90This is stuff that a debugger is supposed to do for you, for free. This should not require code at the application level, but it should be implemented at the tooling level.
Are you running a debugger on a web service in production?
I wouldn't call it a "debugger", but plenty of people run an instrumentation agent like New Relic or AppDynamics that records tracing information on their production web services with little or even zero modification to their application code.