Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracing: Structured logging, but better

andydote.co.uk

Post reply on HN
Print view View original

71–80 of 130 posts

Re: Tracing: Structured logging, but better

#71

Post by FridgeSeal »

crabbone wrote: post #25

> The second problem with writing logs to stdout Who on Earth does that? Logs are almost always written to stderr... In part to prevent other problems author is talking about (eg. mixing with the output generated by the application). I don't understand why this has to be either or... If you store the trace output somewhere you get a log... (let's call it "un-annotated" log, since trace won't have the human-readable m…

I do, as does everyone at my work? Along with basically everyone I’ve ever worked with, ever?

Like, I develop cli apps, so like, what else would go to stdout that you suppose will interfere?

Re: Tracing: Structured logging, but better

#72

Post by nithril »

jeffbee wrote: post #6

This is a great article because everyone should understand the similarity between logging and tracing. One thing worth pondering though is the differences in cost. If I am not planning to centrally collect and index informational logs, free-form text logging is extremely cheap. Even a complex log line with formatted strings and numbers can be emitted in Off-the-shelf tracing libraries on the other hand are pretty exp…

It is actually simpler to conceptualize the difference, one is stateless, the other one is stateful.

Actually structured logging exists since years like in Java https://github.com/logfellow/logstash-logback-encoder

Re: Tracing: Structured logging, but better

#73

Post by Hermitian909 »

layer8 wrote: post #42

> Log Levels are meaningless. Is a log line debug, info, warning, error, fatal, or some other shade in between? I partly agree and disagree. In terms of severity, there are only three levels: – info: not a problem – warning: potential problem – error: actual problem (operational failure) Other levels like “debug” are not about severity, but about level of detail. In addition, something that is an error in a subcompon…

The OP is wrong, log levels are very valuable if you leverage them.

Here's a classic problem as an illustration: The storage cost of your logs is really prohibitive. You would like to cut out some of your logs from storage but cannot lower retention below some threshold (say 2 weeks maybe). For this example, assume that tracing is also enabled and every log has a traceId

A good answer is to run a compaction job that inspects each trace. If it contains an error preserve it. Remove X% of all other traces.

Log levels make the ergonomics for this excellent and it can save millions of dollars a year at sufficient scale.

Re: Tracing: Structured logging, but better

#74

Post by jen20 »

phillipcarter wrote: post #21
candiddevmike wrote: post #18

Earlier quoted context omitted.

That might be dependent on the library then, there isn't an official OTel Go logging library yet. Seems you have to add the trace ID exemplars manually too

Go is behind several of the languages in OTel right now. Just a consequence of a very difficult implementation and its load-bearing nature as being the language (and library) of choice for CNCF infrastructure. If you use Java or .NET, for example, it's quite fleshed out.

One would hope that there will not _be_ an Open Telemetry logging library for Go. Unlike last time there was a thread about this, there is now a standard - `slog` in the stdlib.

Re: Tracing: Structured logging, but better

#75

Post by jameshart »

thangalin wrote: post #48
zoogeny wrote: post #23

One thing about logging and tracing is the inevitable cost (in real money). I love observability probably more than most. And my initial reaction to this article is the obvious: why not both? In fact, I tend to think more in terms of "events" when writing both logs and tracing code. How that event is notified, stored, transmitted, etc. is in some ways divorced from the activity. I don't care if it is going to stdout,…

> In fact, I tend to think more in terms of "events" when writing both logs and tracing code. They are events[1]. For my text editor, KeenWrite, events can be logged either to the console when run from the command-line or displayed in a dialog when running in GUI mode. By changing "logger.log()" statements to "event.publish()" statements, a number of practical benefits are realized, including: * Decoupled logging imp…

But events that another system relies upon are now an API. Be careful not to lock together things that are only superficially similar, as it affects your ability to change them independently.

Re: Tracing: Structured logging, but better

#76

Post by jameshart »

zoogeny wrote: post #23

One thing about logging and tracing is the inevitable cost (in real money). I love observability probably more than most. And my initial reaction to this article is the obvious: why not both? In fact, I tend to think more in terms of "events" when writing both logs and tracing code. How that event is notified, stored, transmitted, etc. is in some ways divorced from the activity. I don't care if it is going to stdout,…

Observability costs feel high when everything’s working fine. When something snaps and everything is down and you need to know why in a hurry… those observability premiums you’ve been paying all along can pay off fast.

Re: Tracing: Structured logging, but better

#77

Post by thangalin »

jameshart wrote: post #75
thangalin wrote: post #48

Earlier quoted context omitted.

> In fact, I tend to think more in terms of "events" when writing both logs and tracing code. They are events[1]. For my text editor, KeenWrite, events can be logged either to the console when run from the command-line or displayed in a dialog when running in GUI mode. By changing "logger.log()" statements to "event.publish()" statements, a number of practical benefits are realized, including: * Decoupled logging imp…

But events that another system relies upon are now an API . Be careful not to lock together things that are only superficially similar, as it affects your ability to change them independently.

Architecturally, the decoupling works as follows:

    Event -> Bus -> UI Subscriber -> Dialog (table)
    Event -> Bus -> Log Subscriber -> Console (text)
    Event -> Bus -> D-Bus Subscriber -> Relay -> D-Bus -> Publish (TCP/IP)
With D-Bus, published messages are versioned, allowing for API changes without breaking third-party consumers. The D-Bus Subscriber provides a layer of isolation between the application and the published messages so that the two can vary independently.

Re: Tracing: Structured logging, but better

#78

Post by SkyPuncher »

layer8 wrote: post #42

> Log Levels are meaningless. Is a log line debug, info, warning, error, fatal, or some other shade in between? I partly agree and disagree. In terms of severity, there are only three levels: – info: not a problem – warning: potential problem – error: actual problem (operational failure) Other levels like “debug” are not about severity, but about level of detail. In addition, something that is an error in a subcompon…

I agree with your premise, but do consider debug to be a fourth level.

Info is things like “processing X”

Debug is things like “variable is Y” or “made it to this point”

Post reply on HN