Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracing: Structured logging, but better

andydote.co.uk

Post reply on HN
Print view View original

61–70 of 130 posts

Re: Tracing: Structured logging, but better

#61

Post by phillipcarter »

hardwaresofton wrote: post #59

I think there's an alternate universe out there where: - we collectively realized that logs, events, traces, metrics, and errors are actually all just logs - we agreed on a single format that encapsulated all that information in a structured manner - we built firehose/stream processing tooling to provide modern o11y creature comforts I can't tell if that universe is better than this one, or worse.

That's more or less the model Honeycomb uses. Every signal type is just a structured event. Reality is a bit messier, though. In particular, metrics are the oddball in this world and required a lot of work to make economical.

Re: Tracing: Structured logging, but better

#62

Post by dalyons »

hardwaresofton wrote: post #59

I think there's an alternate universe out there where: - we collectively realized that logs, events, traces, metrics, and errors are actually all just logs - we agreed on a single format that encapsulated all that information in a structured manner - we built firehose/stream processing tooling to provide modern o11y creature comforts I can't tell if that universe is better than this one, or worse.

Is that really an alternate universe? That’s the universe that splunk and friends are selling, everything’s a log. It’s really expensive.

Re: Tracing: Structured logging, but better

#63

Post by andrewstuart2 »

hardwaresofton wrote: post #59

I think there's an alternate universe out there where: - we collectively realized that logs, events, traces, metrics, and errors are actually all just logs - we agreed on a single format that encapsulated all that information in a structured manner - we built firehose/stream processing tooling to provide modern o11y creature comforts I can't tell if that universe is better than this one, or worse.

Traces are just distributed "logs" (in the data structure sense; data ordered only by its appearance in something) where you also pass around the tiniest bit of correlation context between apps. Traces are structured, timestamped, and can be indexed into much more debug-friendly structures like a call tree. But you could just as easily ignore all the data and print them out in streaming sorted order without any correlation.

Honestly it sounds like you're pitching opentelemetry/otlp but where you only trace and leave all the other bits for later inside your opentelemetry collector, which can turn traces into metrics or traces into logs.

Re: Tracing: Structured logging, but better

#64

Post by dalyons »

crabbone wrote: post #25

> The second problem with writing logs to stdout Who on Earth does that? Logs are almost always written to stderr... In part to prevent other problems author is talking about (eg. mixing with the output generated by the application). I don't understand why this has to be either or... If you store the trace output somewhere you get a log... (let's call it "un-annotated" log, since trace won't have the human-readable m…

Being doing it for decade+, ever since the 12 factor app concept became popular. It’s way more common imho for web apps than stderr logging.

Re: Tracing: Structured logging, but better

#65

Post by mrkeen »

rewmie wrote: post #52
mrkeen wrote: post #47

> If you’re writing log statements, you’re doing it wrong. I too use this bait statement. Then I follow it up with (the short version): 1) Rewrite your log statements so that they're machine readable 2) Prove they're machine-readable by having the down-stream services read them instead of the REST call you would have otherwise sent. 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for y…

> 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for you. I don't think this is a reasonable statement. There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka. Bringing up Kafka sounds like a case of a solution looking for a problem.

> There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka.

What are they? Because admittedly I've lost a little love for the operational side of Kafka, and I wish the client-side were a little "dumber", so I could match it better to my uses cases.

Re: Tracing: Structured logging, but better

#66

Post by mrkeen »

bowsamic wrote: post #53
mrkeen wrote: post #47

> If you’re writing log statements, you’re doing it wrong. I too use this bait statement. Then I follow it up with (the short version): 1) Rewrite your log statements so that they're machine readable 2) Prove they're machine-readable by having the down-stream services read them instead of the REST call you would have otherwise sent. 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for y…

How to get me to leave your company 101

I did write a pretty glib description of what to do ;)

That said, I've had conflicts with a previous team-mate about this. He couldn't wrap his head around Kafka being a source of truth. But when I asked him whether he'd trust our Kafka or our Postgres if they disagreed, he conceded that he'd believe Kafka's side of things.

Re: Tracing: Structured logging, but better

#67

Post by skybrian »

How would a hobbyist programmer get started with tracing for a simple web app? Where do the traces end up and how do I query it? Can tracing be used in a development environment?

Context: the last thing I wrote used Deno and Deno Deploy.

Re: Tracing: Structured logging, but better

#68

Post by perpil »

I was recently musing about the 2 different types of logs:

1. application logs, emitted multiple times per request and serve as breadcrumbs

2. request logs emitted once per request and include latencies, counters and metadata about the request and response

The application logs were useless to me except during development. However the request logs I could run aggregations on which made them far more useful for answering questions. What the author explains very well is that the problem with application logs is they aren't very human-readable which is where visualizing a request with tracing shines. If you don't have tracing, creating request logs will get you most of the way there, it's certainly better than application logs. https://speedrun.nobackspacecrew.com/blog/2023/09/08/logging...

Re: Tracing: Structured logging, but better

#69

Post by curioussavage »

skybrian wrote: post #67

How would a hobbyist programmer get started with tracing for a simple web app? Where do the traces end up and how do I query it? Can tracing be used in a development environment? Context: the last thing I wrote used Deno and Deno Deploy.

Just install opentelemetry libs. I found this example with a quick search: https://dev.to/grunet/leveraging-opentelemetry-in-deno-45bj

opentelemetry has a service you can run that will collect the telemetry data and you can export it to something like prometheus which can store it and let you query it. Example here https://github.com/open-telemetry/opentelemetry-collector-co...

Typically in dev environments trace spans are just emitted to stdout just like logs. I sometimes turn that off too though because it gets noisy.

Re: Tracing: Structured logging, but better

#70

Post by benreesman »

As a historical critic of Rust-mania (and if I’m honest, kind of an asshole about it too many times, fail), I’ve recently bumped into stuff like tokio-tracing, eyre, tokio-console, and some others.

And while my historical gripes are largely still the status quo: stack traces in multi-threaded, evented/async code that actually show real line numbers? Span-based tracing that makes concurrent introspection possible by default?

I’m in. I apologize for everything bad I ever said and don’t care whatever other annoying thing.

That’s the whole show. Unless it deletes my hard drive I don’t really care about anything else by comparison.

Post reply on HN