Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Five Levels of Logging

aib42.net

Post reply on HN
Print view View original

41–50 of 61 posts

Re: The Five Levels of Logging

#41

Post by Rapzid »

nicolaslem wrote: post #37
sagichmal wrote: post #29

Earlier quoted context omitted.

Any exception that bubbles all the way up to a request-scoped catch block indicates a bug in your handler logic. It should probably crash your program, or at least be reported as a structured event to your exception tracking system. In either case, logger.Info is sufficient to record this event in the application logs.

Why should exception tracking and logging be two separated things? They perform the same job, an exception tracking system is just another way to present log records. The real problem is that people still associate logs with lines of text in a file, whereas most logging frameworks create full objects containing a ton of information[0]. Once you stop considering logs as text, exception tracking and logging really beco…

Exactly. As a subset of monitoring, it can probably be thought of as eventing; like all monitoring. What's a metric? A measurement event. Etc.

Perhaps a name change is in order; Event Tracing for Windows anyone? Eventing.. Event Tracing. IDK.

Re: The Five Levels of Logging

#42

Post by sagichmal »

philwelch wrote: post #35
sagichmal wrote: post #28

Earlier quoted context omitted.

If you're parsing your logs with a machine for triage like you describe, you should be doing structured logging, not string logging.

And the structure will include a log level to enable this kind of classification.

Sure, but then we're not talking logger.Info: it's a difference in kind.

Re: The Five Levels of Logging

#43

Post by sagichmal »

nicolaslem wrote: post #37
sagichmal wrote: post #29

Earlier quoted context omitted.

Any exception that bubbles all the way up to a request-scoped catch block indicates a bug in your handler logic. It should probably crash your program, or at least be reported as a structured event to your exception tracking system. In either case, logger.Info is sufficient to record this event in the application logs.

Why should exception tracking and logging be two separated things? They perform the same job, an exception tracking system is just another way to present log records. The real problem is that people still associate logs with lines of text in a file, whereas most logging frameworks create full objects containing a ton of information[0]. Once you stop considering logs as text, exception tracking and logging really beco…

Conceptually they are very similar, in the same way that metric data and log data are also very similar. But we don't have good technological solutions to support muxing all streams of observability data together and dealing with them efficiently, either in transport or in storage/query.

Good program architecture in 2018 demands some degree of compromise. In my experience it's best to separate console logging (as in this article) from structured logging, from metrics, from distributed tracing, from exception tracking.

Re: The Five Levels of Logging

#44

Post by pfranz »

ath0 wrote: post #19

I support these 5 levels of logging, but given the discussion below I also would like to highlight a lesson from an SRE talk I once attended[citation needed] that really there are only three levels of alert: 1. DEBUG - Look at this when debugging, otherwise no one will ever see it. 2. TICKET - Open a ticket and have someone on your team look at this when it gets to the top of your queue. 3. PAGER - Drop what you're d…

It's nice to have articles like this to give structure to the conversation, but there's no single solution. Python has default levels of; debug, info, warning, error, and critical, but doesn't really give any guidance to what each means. At one company we tried to create some structure, but 3rd party libraries would flood messages at the "wrong" logging level.

I like these 5-levels and the 3-levels you talk about seem like a paired down version that reenforces what each level means.

To illustrate the "no single solution," we tried to use logging for command line output, neither of those systems really has a level for that. We probably should have created a new level between INFO and WARNING because of chatty libraries.

In practice I've found the DEBUG and TRACE are usually best added to code when troubleshooting. When you're writing the code you don't have a good idea what info you'd need when troubleshooting, so it's overly verbose but still missing what you'd want. When trying to stash all logging either locally or centrally (because adding logging and re-running isn't always an option) it can affect performance.

Re: The Five Levels of Logging

#45

Post by blaisio »

I really believe only two levels are necessary most of the time: error, and info. Error is for anything that a dev should be notified about (AKA if you send an error log message, assume someone will be notified about it immediately), and info is for any other useful debugging information, like "we got a request for /foo/bar at 10 pm". I also think structured logging is absolutely essential. Log levels are really just a way to get some of the benefits of structured logging.

Re: The Five Levels of Logging

#46

Post by pfranz »

xtracto wrote: post #10

This article reminded me of a writing that an ex-colleague of mine did at a previous company: https://nerds.kueski.com/better-logging/ Everybody developer knows that they should include logging in their systems, however how and what to log is not something so well documented. When you start working in a system (i.e. at a startup) you put the odd log here and there, without too much attention. However, as your system(…

What's nice about something like Elasticsearch is if you categorize your logging levels you can more easily drop logs of a certain level at varying times. Debug/verbose/trace is a lot less useful after, say, 24hrs and takes up a lot of space while Fatal errors should be rare and probably want to be tracked year-over-year or between releases.

Re: The Five Levels of Logging

#47

Post by jschwartzi »

pfranz wrote: post #44
ath0 wrote: post #19

I support these 5 levels of logging, but given the discussion below I also would like to highlight a lesson from an SRE talk I once attended[citation needed] that really there are only three levels of alert: 1. DEBUG - Look at this when debugging, otherwise no one will ever see it. 2. TICKET - Open a ticket and have someone on your team look at this when it gets to the top of your queue. 3. PAGER - Drop what you're d…

It's nice to have articles like this to give structure to the conversation, but there's no single solution. Python has default levels of; debug, info, warning, error, and critical, but doesn't really give any guidance to what each means. At one company we tried to create some structure, but 3rd party libraries would flood messages at the "wrong" logging level. I like these 5-levels and the 3-levels you talk about see…

I think a good log system should give you the ability to store everything but filter quickly. So for example I'll set up my systems to store everything down to LOG_DEBUG because I might want that information. But when I'm actually looking at the logs I like being able to exclude the DEBUG messages and look at the general behavior, then drill down when I think I understand what the sequence of events is. Good log systems should support this without making me filter as I'm storing the data.

Also in UNIX systems I wish there was a way to generically pipe the output of a command to the system log while preserving the return code in case the command exits. In BASH you can configure the pipe behavior to propagate errors to the calling shell, but in normal shells the last return code is the one that gets stored, which is not usually what you want. It would be really cool if the logger utility captured return codes from upstream and returned them so that I can use it in the antiquated SystemV init scripts I have to maintain.

Re: The Five Levels of Logging

#49

Post by hinkley »

Rapzid wrote: post #39
hinkley wrote: post #30

Earlier quoted context omitted.

Logging is one of the weirder cross cutting concerns, for many reasons but especially including the one you mention. We had to roll back an upgrade last month because one nonfatal (but still pretty bad) problem was generating a ridiculous amount of log entries. We had no way to turn it off or throttle it, so we rolled back. I have fantasized for years about a programming language with a small runtime that supports in…

Check out log points: https://code.visualstudio.com/docs/editor/debugging#_logpoin...

It's buried but my IDE has a similar feature.

I meant support for running systems, not local development.

Re: The Five Levels of Logging

#50

Post by Rapzid »

hinkley wrote: post #49
Rapzid wrote: post #39

Earlier quoted context omitted.

Check out log points: https://code.visualstudio.com/docs/editor/debugging#_logpoin...

It's buried but my IDE has a similar feature. I meant support for running systems, not local development.

"Logpoints are especially useful for injecting logging while debugging production servers that cannot be paused or stopped."

There is a lot more going on with them than you might realize. Not sure how coupled it is to the nodejs/kubernetes/Azure stuff they have built into VSCode, but it seems like a working example of the future. Sorta like time travel debugging in node-chakracore.

Windows has this facility called EWT that lets you setup all sorts of eventing/tracing that is actually not present(the calls) in the code unless the EWT facility injects the code necessary to enable it into the running process. This allows you to dynamically enable and disable event sources based on stuff like.. whether or not something is currently subscribing to it. Worth looking into just for the sheer amazement of it if you are not a Windows developer and familiar with it.

Post reply on HN