Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracing: Structured logging, but better

andydote.co.uk

Post reply on HN
Print view View original

51–60 of 130 posts

Re: Tracing: Structured logging, but better

#51

Post by layer8 »

BillinghamJ wrote: post #45
layer8 wrote: post #42

> Log Levels are meaningless. Is a log line debug, info, warning, error, fatal, or some other shade in between? I partly agree and disagree. In terms of severity, there are only three levels: – info: not a problem – warning: potential problem – error: actual problem (operational failure) Other levels like “debug” are not about severity, but about level of detail. In addition, something that is an error in a subcompon…

I tend to think of "warning" as - "something unexpected happened, but it was handled safely" And then "error" as - "things are not okay, a developer is going to need to intervene" And errors then split roughly between "must be fixed sometime", and "must be fixed now/ASAP"

> I tend to think of "warning" as - "something unexpected happened, but it was handled safely"

It was handled safely at the level where it occurred, but because it was unusual/unexpected, the underlying cause may cause issues later on or higher up.

If one were sure it would 100% not indicate any issue, one wouldn’t need to warn about it.

Re: Tracing: Structured logging, but better

#52

Post by rewmie »

mrkeen wrote: post #47

> If you’re writing log statements, you’re doing it wrong. I too use this bait statement. Then I follow it up with (the short version): 1) Rewrite your log statements so that they're machine readable 2) Prove they're machine-readable by having the down-stream services read them instead of the REST call you would have otherwise sent. 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for y…

> 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for you.

I don't think this is a reasonable statement. There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka. Bringing up Kafka sounds like a case of a solution looking for a problem.

Re: Tracing: Structured logging, but better

#53

Post by bowsamic »

mrkeen wrote: post #47

> If you’re writing log statements, you’re doing it wrong. I too use this bait statement. Then I follow it up with (the short version): 1) Rewrite your log statements so that they're machine readable 2) Prove they're machine-readable by having the down-stream services read them instead of the REST call you would have otherwise sent. 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for y…

How to get me to leave your company 101

Re: Tracing: Structured logging, but better

#54

Post by layer8 »

beachy wrote: post #49
layer8 wrote: post #42

> Log Levels are meaningless. Is a log line debug, info, warning, error, fatal, or some other shade in between? I partly agree and disagree. In terms of severity, there are only three levels: – info: not a problem – warning: potential problem – error: actual problem (operational failure) Other levels like “debug” are not about severity, but about level of detail. In addition, something that is an error in a subcompon…

> In addition, something that is an error in a subcomponent may only be a warning or even just an info on the level of the superordinate component. Or, keep it simple. - error means someone is alerted urgently to look at the problem - warning means someone should be looking into it eventually, with a view to reclassifying as info/debug or resolving it. IMO many people don't care much about their logs, until the shit…

> - error means someone is alerted urgently to look at the problem

The issue is that the code that encounters the problem may not have the knowledge/context to decide whether it warrants alerting. The code higher up that does have the knowledge, on the other hand, often doesn’t have the lower-level information that is useful to have in the log for analyzing the failure. So how do you link the two? When you write modular code that minimizes assumptions about its context, that situation is a common occurrence.

Re: Tracing: Structured logging, but better

#55

Post by hyperpape »

jeffbee wrote: post #6

This is a great article because everyone should understand the similarity between logging and tracing. One thing worth pondering though is the differences in cost. If I am not planning to centrally collect and index informational logs, free-form text logging is extremely cheap. Even a complex log line with formatted strings and numbers can be emitted in Off-the-shelf tracing libraries on the other hand are pretty exp…

I don't generally disagree, but using json for structured logs is a growing thing as well.

Re: Tracing: Structured logging, but better

#56

Post by ahoka »

rewmie wrote: post #52
mrkeen wrote: post #47

> If you’re writing log statements, you’re doing it wrong. I too use this bait statement. Then I follow it up with (the short version): 1) Rewrite your log statements so that they're machine readable 2) Prove they're machine-readable by having the down-stream services read them instead of the REST call you would have otherwise sent. 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for y…

> 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for you. I don't think this is a reasonable statement. There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka. Bringing up Kafka sounds like a case of a solution looking for a problem.

I think OP meant event sourcing.

Re: Tracing: Structured logging, but better

#57

Post by rewmie »

ahoka wrote: post #56
rewmie wrote: post #52

Earlier quoted context omitted.

> 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for you. I don't think this is a reasonable statement. There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka. Bringing up Kafka sounds like a case of a solution looking for a problem.

I think OP meant event sourcing.

> I think OP meant event sourcing.

That is really besides the point. Logging and tracing have always been fundamentally event sourcing, but that never forced anyone ever at all to onboard onto freaking Kafka of all event streaming/messaging platforms.

This blend of suggestion sounds an awful lot like resume driven development instead of actually putting together a logging service.

Re: Tracing: Structured logging, but better

#58

Post by WatchDog »

alkonaut wrote: post #40

I like a log to read like a book if it’s the result of a task taking a finite time, such as for example an installation, a compilation, a loading of a browser page or similar. Users are going to look into it for clues about what happened and they a) aren’t always related to those who wrote the tools b) don’t have access to the source code or any special log analytics/querying tools. That’s when you want a log and tha…

It’s a good distinction to make, logging for client based systems, is essentially UI design.

For a web app, serving lots of concurrent users, they are essentially unreadable without tools, so you may as well optimise the logs for tool based consumption.

Re: Tracing: Structured logging, but better

#59

Post by hardwaresofton »

I think there's an alternate universe out there where:

- we collectively realized that logs, events, traces, metrics, and errors are actually all just logs

- we agreed on a single format that encapsulated all that information in a structured manner

- we built firehose/stream processing tooling to provide modern o11y creature comforts

I can't tell if that universe is better than this one, or worse.

Re: Tracing: Structured logging, but better

#60

Post by beachy »

layer8 wrote: post #54
beachy wrote: post #49

Earlier quoted context omitted.

> In addition, something that is an error in a subcomponent may only be a warning or even just an info on the level of the superordinate component. Or, keep it simple. - error means someone is alerted urgently to look at the problem - warning means someone should be looking into it eventually, with a view to reclassifying as info/debug or resolving it. IMO many people don't care much about their logs, until the shit…

> - error means someone is alerted urgently to look at the problem The issue is that the code that encounters the problem may not have the knowledge/context to decide whether it warrants alerting. The code higher up that does have the knowledge, on the other hand, often doesn’t have the lower-level information that is useful to have in the log for analyzing the failure. So how do you link the two? When you write modu…

If the code detecting the error is a library/subordinate service then the same rule can be followed - should this be immediately brought to a human's attention?

The answer for a library will often be no, since the library doesn't "have the knowledge/context to decide whether it warrants alerting".

So in that case the library can log as info, and leave it to the caller to log as error if warranted (after learning about the error from return code/http status etc.).

When investigating the error, the human has access to the info details from the subordinate service.

Post reply on HN