Log level 'error' should mean that something needs to be fixed • Hacker News Board

Re: Log level 'error' should mean that something needs to be fixed

#81

Post by ivan_gammel » Sat, Dec 20, 2025, 6:34 PM UTC

mewpmewp2 wrote: post #67

eterm wrote: post #59

Earlier quoted context omitted.

This might be controversial, but I'd say if it's fine after a retry, then it doesn't need a warning. Because what I'd want to know is how often does it fail , which is a metric not a log. So expose as a metric not a log. If feeding logs into datadog or similar is the only way you're collecting metrics, then you aren't treating your observablity with the respect it deserves. Put in real counters so you're not just rea…

> If the third party being down has a knock-on effect to your own system functionality / uptime, then it needs to be a warning or error, but you should also put in the backlog a ticket to de-couple your uptime from that third-party, be it retries, queues, or other mitigations ( alternate providers? ). How do you define uptime? What if e.g. it's a social login / data linking and that provider is down? You could have m…

You may log that or count failures in some metric, but the correct answer is to have a health check on third party service and an alert when that service is down. Logs may help to understand the nature of the incident, but they are not the channel through which you are informed about such problems.

The different issue is when third party broke the contract, so suddenly you get a lot of 4xx or 5xx responses, likely unrecoverable. Then you get ERROR level messages in the log (because it’s unexpected problem) and an alert when there’s a spike.

Re: Log level 'error' should mean that something needs to be fixed

#82

Post by hinkley » Sat, Dec 20, 2025, 6:37 PM UTC

layer8 wrote: post #76

> When implementing logging, it's important to distinguish between an error from the perspective of an individual operation and an error from the perspective of the overall program or system. Individual operations may well experience errors that are not error level log events for the overall program. You could say that an operation error is anything that prevents an operation from completing successfully, while a pro…

Log4j has the ability to filter log levels by subject matter for twenty years. In Java you end up having to use that a lot for this reason.

Re: Log level 'error' should mean that something needs to be fixed

#83

Post by layer8 » Sat, Dec 20, 2025, 6:38 PM UTC

ivan_gammel wrote: post #78

layer8 wrote: post #76

> When implementing logging, it's important to distinguish between an error from the perspective of an individual operation and an error from the perspective of the overall program or system. Individual operations may well experience errors that are not error level log events for the overall program. You could say that an operation error is anything that prevents an operation from completing successfully, while a pro…

Libraries should not log on levels above DEBUG, period. If there’s something worthy for reporting on higher levels, pass this information to client code, either as an event, or as an exception or error code.

From a code modularization point of view, there shouldn’t really be much of a difference between programs and libraries. A program is just a library with a different calling convention. I like to structure programs such that their actual functionality could be reused as a library in another program.

This is difficult to reconcile with libraries only logging on a debug level.

Re: Log level 'error' should mean that something needs to be fixed

#84

Post by georgefrowny » Sat, Dec 20, 2025, 6:38 PM UTC

Easy to say, but there's "yes we know this is wrong but this will have to do for now" and "we don't expect to see this in real life unless something has gone sideways".

Re: Log level 'error' should mean that something needs to be fixed

#85

Post by hinkley » Sat, Dec 20, 2025, 6:40 PM UTC

zamadatix wrote: post #42

greatgib wrote: post #28

Earlier quoted context omitted.

The point is that if your program itself take note of the error from the library it is ok. You, as the program owner, can decide what to do with it (error log or not). But if you are the SMTP library and that you unilaterally log that as an error. That is an issue.

The counterpoint made above is while what you describe is indeed the way the author likes to see it that doesn't explain why "an error is something which failed that the program was unable to fix automatically" is supposed to be any less valid a way to see it. I.e. should error be defined as "the program was unable to complete the task you told it to do" or only "things which could have worked but you need to explici…

Any robust system isn’t going to rely on reading logs to figure out what to do about undelivered email anyway. If you’re doing logistics the failure to send an order confirmation needs to show up in your data model in some manner. Managing your application or business by logs is amateur hour.

There’s a whole industry of “we’ll manage them for you” which is just enabling dysfunction.

Re: Log level 'error' should mean that something needs to be fixed

#86

Post by ivan_gammel » Sat, Dec 20, 2025, 6:40 PM UTC

layer8 wrote: post #83

ivan_gammel wrote: post #78

Earlier quoted context omitted.

Libraries should not log on levels above DEBUG, period. If there’s something worthy for reporting on higher levels, pass this information to client code, either as an event, or as an exception or error code.

From a code modularization point of view, there shouldn’t really be much of a difference between programs and libraries. A program is just a library with a different calling convention. I like to structure programs such that their actual functionality could be reused as a library in another program. This is difficult to reconcile with libraries only logging on a debug level.

The main difference is that library is not aware of the context of the execution of the code, so cannot decide, whether the problem is expected, recoverable or severe.

Re: Log level 'error' should mean that something needs to be fixed

#87

Post by 1123581321 » Sat, Dec 20, 2025, 6:42 PM UTC

rwmj wrote: post #79

1123581321 wrote: post #39

Earlier quoted context omitted.

Can you please explain this? That sounds like identifying bugs but not fixing them but I realize you don’t mean that. One hopes the context information in the error will make it actionable when it occurs, never completely successfully, of course.

Here's an example of a bug that I filed about non-actionable error messages: https://github.com/karmab/kcli/issues/456 The first error message was "No usable public key found, which is required for the deployment" which doesn't tell me what I have to do to correct the problem. Nothing about even where it's looking for keys, what is supposed to create the key or how I am supposed to create the key. There are other exa…

That makes sense and good examples; thanks.

At work, I can think of cases where we error when data mismatches between two systems. It’s almost always the fault of system B but we present the mismatch error neutrally. Experienced developers just know to fix B but we shouldn’t rely on that.

Re: Log level 'error' should mean that something needs to be fixed

#88

Post by 0x696C6961 » Sat, Dec 20, 2025, 6:42 PM UTC

layer8 wrote: post #76

> When implementing logging, it's important to distinguish between an error from the perspective of an individual operation and an error from the perspective of the overall program or system. Individual operations may well experience errors that are not error level log events for the overall program. You could say that an operation error is anything that prevents an operation from completing successfully, while a pro…

Libraries should not log, instead they should allow registering hooks which get called with errors and debug info.

Re: Log level 'error' should mean that something needs to be fixed

#89

Post by oofbey » Sat, Dec 20, 2025, 6:43 PM UTC

georgefrowny wrote: post #84

Easy to say, but there's "yes we know this is wrong but this will have to do for now" and "we don't expect to see this in real life unless something has gone sideways".

At scale the rare events start to happen reliably. Hardware failures almost certainly cause ERROR conditions. Network glitches.

Our production system pages oncall for any errors. At night it will only wake somebody up for a whole bunch of errors. This discipline forces us to take a look at every ERROR and decide if it is spurious and out of our control or something we can deal with. At some point our production system will reach a scale where there are errors logged constantly and this strategy Durant make sense any more. But for now it helps keep our system clean.

Re: Log level 'error' should mean that something needs to be fixed

#90

Post by [deleted] » Sat, Dec 20, 2025, 6:44 PM UTC

layer8 wrote: post #83

ivan_gammel wrote: post #78

Earlier quoted context omitted.

Libraries should not log on levels above DEBUG, period. If there’s something worthy for reporting on higher levels, pass this information to client code, either as an event, or as an exception or error code.

From a code modularization point of view, there shouldn’t really be much of a difference between programs and libraries. A program is just a library with a different calling convention. I like to structure programs such that their actual functionality could be reused as a library in another program. This is difficult to reconcile with libraries only logging on a debug level.

[deleted]