Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log level 'error' should mean that something needs to be fixed

utcc.utoronto.ca

Post reply on HN
Print view View original

181–190 of 313 posts

Re: Log level 'error' should mean that something needs to be fixed

#181

Post by awesome_dude »

Too wrote: post #153
MobiusHorizons wrote: post #122

Earlier quoted context omitted.

What you are proposing sounds like a nightmare to debug. The high level perspective of the operation is of course valuable for determining if an investigation is necessary, but the low level perspective in the library code is almost always where the relevant details are hiding. Not logging these details means you are in the dark about anything your abstractions are hiding from higher level code (which is usually a lo…

Simple: include those relevant details in the exceptions instead of hiding them.

At the extreme end: If my Javascript frontend is being told about a database configuration error happening in the backend when a call with specific parameters is made - that is a SERIOUS security problem.

Errors are massaged for the reader - a database access library will know that a DNS error occurred and that is (the first step for debugging) why it cannot connect to the specified datastore. The service layer caller does not need to know that there is a DNS error, it just needs to know that the specified datastore is uncontactable (and then it can move on to the approriate resilience strategy, retry that same datastore, fallback to a different datastore, or tell the API that it cannot complete the call at all).

The caller can then decide what to do (typically say "Well, I tried, but nothing's happening, have yourself a merry 500)

It makes no sense for the Service level to know the details of why the database access layer could not connect, no more than it makes any sense for the database access layer to know why there is a DNS configuration error - the database access just needs to log the reasons (for humans to investigate), and tell the caller (the service layer) that it could not do the task it was asked to do.

If the service layer is told that the database access layer encountered a DNS problem, what is it going to do?

Nothing, the best it can do is log (tell the humans monitoring it) that a DB access call (to a specific DB service layer) failed, and try something else, which is a generic strategy, one that applies to a host of errors that the database call could return.

Re: Log level 'error' should mean that something needs to be fixed

#182

Post by Arrowmaster »

adrianmonk wrote: post #130
colechristensen wrote: post #43

Earlier quoted context omitted.

How about this: - An error is an event that someone should act on. Not necessarily you. But if it's not an event that ever needs the attention of a person then the severity is less than an error. Examples: Invalid credentials. HTTP 404 - Not Found, HTTP 403 Forbidden, (all of the HTTP 400s, by definition) It's not my problem as a site owner if one of my users entered the wrong URL or typed their password wrong, but i…

> An error is an event that someone should act on. Not necessarily you. Personally, I'd further qualify that. It should be logged as an error if the person who reads the logs would be responsible for fixing it. Suppose you run a photo gallery web site. If a user uploads a corrupt JPEG, and the server detects that it's corrupt and rejects it, then someone needs to do something, but from the point of view of the person…

Counter argument. How do you know the user uploaded a corrupted image and it didn't get corrupted by your internet connection, server hardware, or a bug in your software stack?

You cannot accurately assign responsibility until you understand the problem.

Re: Log level 'error' should mean that something needs to be fixed

#183

Post by Xss3 »

Some programs are error resistant and need an additional level: Fatal.

A warning can be ignored safely. Warnings may be 'debugging enabled, results cannot be certified' or something similar.

An error should not be ignored, an operation is failing, data loss may be occurring, etc.

Some users may be okay with that data loss or failing operation. Maybe it isnt important to them. If the program continues and does not error in the parts that matter to the user, then they can ignore it, but it is still objectively an error occurring.

A fatal message cannot be ignored, the system has crashed. Its the last thing you see before shutdown is attempted.

Re: Log level 'error' should mean that something needs to be fixed

#184

Post by ivan_gammel »

dpark wrote: post #176
ivan_gammel wrote: post #138

Earlier quoted context omitted.

In such scenarios it makes sense to give clients an opportunity to react on such conditions programmatically, so just logging is wrong choice and if there’s a call back to client, client can decide whether to log it and how.

I’ve written code that followed this model, but it almost always just maps to logging anyway, and the rest of the time it’s narrow options presented in the callback. e.g. Retry vs wait vs abort. It’s very rarely realistic that a client would code up meaningful paths for every possible failure mode in a library. These callbacks are usually reserved for expected conditions.

> almost always just maps to logging anyway

Yes, that’s the point. You log it until you encounter it for the first time, then you know more and can do something meaningful. E.g. let’s say you build an API client and library offers callback for HTTP 429. You don’t expect it to happen, so just log the errors in a generic handler in client code, but then after some business logic change you hit 429 for the first time. If library offers you control over what is going to happen next, you may decide how exactly you will retry and what happens to your state in between the attempts. If library just logs and starts retry cycle, you may get a performance hit that will be harder to fix.

Re: Log level 'error' should mean that something needs to be fixed

#185

Post by esrauch »

ivan_gammel wrote: post #157
esrauch wrote: post #152

Earlier quoted context omitted.

It's a nice idea but I've literally never seen it done, so I would be interested if you have examples of major libraries that do this. Abstractly it doesn't really seem to work to me in place of simple logs. One test case here is that your library has existed for a decade and was fast, but Java removed a method that let you make it fast, but you can still run slow without that API. Java the runtime has a flag that th…

> library has existed for a decade >but Java removed a method that let you make it fast, but you can still run slow without that API I’d like to see an example of that, because this is extremely hypothetical scenario. I don’t think any library is so advanced to anticipate such scenarios and write something to log. And of course Java specifically has longer cycle of deprecation and removal. :) As for your second examp…

Protobuf is the example I had in mind. It uses sun.misc.Unsafe which is being removed in upcoming Java releases, but it has a slow fallback path. It logs a warning when it runs if it can tell it's only using the fallback path but the fast path is still available if the application owner set a flag to turn it back on if they want to:

https://github.com/protocolbuffers/protobuf/issues/20760

Java Protobuf also logs a warning now if you can tell you are using gencode old enough that it's covered by a DoS CVE. They actually did a release that broke compatability of the CVE covered gencode but restored it and print a warning in a newer release.

Re: Log level 'error' should mean that something needs to be fixed

#186

Post by [deleted] »

kijin wrote: post #58
AndroTux wrote: post #52

“cannot contact port 25 on ” may very well be a configuration error. How should the program know?

SMTP clients are designed to try again with exponential backoff. If the final attempt fails and your email gets bounced, now that's an error. Until then, it's just a delay, business as usual.

[deleted]

Re: Log level 'error' should mean that something needs to be fixed

#187

Post by usefulcat »

cogman10 wrote: post #132
Too wrote: post #92

Earlier quoted context omitted.

This is why it’s almost always wrong for library functions to log anything, even on ”errors”. Pass the status up through return values or exceptions. As a library author you have no clue as how an application might use it. Multi threading, retry loops and expected failures will turn what’s a significant event in one context into what’s not even worthy of a debug log in another. No rule without exceptions of course, o…

Depending on the language and logging framework, debug/trace logging can be acceptable in a library. But you have to be extra careful to make sure that it's ultimately a no-op. A common problem in Java is someone will drop a log that looks something like this `log.trace("Doing " + foo + " to " + bar);` The problem is, especially in a hot loop, that throw away string concatenation can ultimately be a performance probl…

Ideally a logging library should at least not make it easy to make that kind of mistake.

Re: Log level 'error' should mean that something needs to be fixed

#188

Post by StellarScience »

HendrikHensen wrote: post #149
layer8 wrote: post #76

> When implementing logging, it's important to distinguish between an error from the perspective of an individual operation and an error from the perspective of the overall program or system. Individual operations may well experience errors that are not error level log events for the overall program. You could say that an operation error is anything that prevents an operation from completing successfully, while a pro…

> Should only “top-level” code ever log an error? That can make it difficult to identify the low-level root causes of a top-level failure. Some languages (e.g. Java) include a stack trace when reporting an error, which is extremely useful when logging the error. It shows at exactly which point in the code the error was generated, and what the full call stack was to get there. It's a real shame that "modern" languages…

C++ with Boost has let you grab a stacktrace anywhere in the application for years. But in April 2024 Boost 1.85 added a big new feature: stacktrace from arbitrary exception ( https://www.boost.org/releases/1.85.0/ ), which shows the call stack at the time of the throw. We added it to our codebase, and suddenly errors where exceptions were thrown became orders of magnitude easier to debug.

C++23 added std::tracktrace, but until it includes stacktrace from arbitrary exception, we're sticking with Boost.

Re: Log level 'error' should mean that something needs to be fixed

#189

Post by MobiusHorizons »

Too wrote: post #153
MobiusHorizons wrote: post #122

Earlier quoted context omitted.

What you are proposing sounds like a nightmare to debug. The high level perspective of the operation is of course valuable for determining if an investigation is necessary, but the low level perspective in the library code is almost always where the relevant details are hiding. Not logging these details means you are in the dark about anything your abstractions are hiding from higher level code (which is usually a lo…

Simple: include those relevant details in the exceptions instead of hiding them.

Sometimes you don’t have all the relevant details in scope at the point of error. For instance some recoverable thing might have happened first which exercises a backup path with slightly different data. This is not exception worthy and execution continues. Then maybe some piece of data in this backup path interacts poorly with some other backend causing an error. The exception won’t tell you how you got there, only where you got stuck. Logging can tell you the steps that led up to that, which is useful. Of course you need a way to deal with verbose logs effectively, but such systems aren’t exactly rare these days.

Re: Log level 'error' should mean that something needs to be fixed

#190

Post by dpark »

ivan_gammel wrote: post #184
dpark wrote: post #176

Earlier quoted context omitted.

I’ve written code that followed this model, but it almost always just maps to logging anyway, and the rest of the time it’s narrow options presented in the callback. e.g. Retry vs wait vs abort. It’s very rarely realistic that a client would code up meaningful paths for every possible failure mode in a library. These callbacks are usually reserved for expected conditions.

> almost always just maps to logging anyway Yes, that’s the point. You log it until you encounter it for the first time, then you know more and can do something meaningful. E.g. let’s say you build an API client and library offers callback for HTTP 429. You don’t expect it to happen, so just log the errors in a generic handler in client code, but then after some business logic change you hit 429 for the first time. I…

Defining a callback for every situation where a library might encounter an unexpected condition and pointing them all at the logs seems like a massive waste of time.

I would much prefer a library have sane defaults, reasonable logging, and a way for me to plug in callbacks where needed. Writing On429 and a hundred other functions that just point to Logger.Log is not a good use of time.

Post reply on HN