Log level 'error' should mean that something needs to be fixed • Hacker News Board

Re: Log level 'error' should mean that something needs to be fixed

#41

Post by zbentley » Sat, Dec 20, 2025, 5:23 PM UTC

raldi wrote: post #13

zbentley wrote: post #12

Earlier quoted context omitted.

I wish I lived in a world where that worked. Instead, I live in a world where most downstream service issues (including database failures, network routing misconfigurations, giant cloud provider downtime, and more ordinary internal service downtime) are observed in the error logs of consuming services long before they’re detected by the owners of the downstream service … if they ever are. My rough guess is that 75% o…

But everything you’re describing can be done with metrics and alerts; there’s no need to spam the ERROR loglevel.

My point is that just because those problems can be solved with better telemetry doesn’t mean that is actually done in practice. Most organizations do are much more aware of/sensitive to failures upstream/at the edge than they are in backend services. Once you account for alert fatigue, crappy accountability distribution, and organizational pressures, even the places that do this well often backslide over time.

In brief: drivers don’t obey the speed limit and backend service operators don’t prioritize monitoring. Both groups are supposed to do those things, but they don’t and we should assume they won’t change. As a result, it’s a good idea to wear seatbelts and treat downstream failures as urgent errors in the logs of consuming services.

Re: Log level 'error' should mean that something needs to be fixed

#42

Post by zamadatix » Sat, Dec 20, 2025, 5:23 PM UTC

greatgib wrote: post #28

mfuzzey wrote: post #8

I think it's difficult to say without knowing how the system is deployed and administered. "If a SMTP mailer trying to send email to somewhere logs 'cannot contact port 25 on ', that is not an error in the local system" Maybe or maybe not. If the connection problem is really due to the remote host then that's not the problem of the sender. But maybe the local network interface is down, maybe there's a local firewall…

The point is that if your program itself take note of the error from the library it is ok. You, as the program owner, can decide what to do with it (error log or not). But if you are the SMTP library and that you unilaterally log that as an error. That is an issue.

The counterpoint made above is while what you describe is indeed the way the author likes to see it that doesn't explain why "an error is something which failed that the program was unable to fix automatically" is supposed to be any less valid a way to see it. I.e. should error be defined as "the program was unable to complete the task you told it to do" or only "things which could have worked but you need to explicitly change something locally".

I don't even know how to say whether these definitions are right or wrong, it's just whatever you feel like it should be. The important thing is what your program logs should be documented somewhere, the next most important thing is that your log levels are self consistent and follow some sort of logic, and that I would have done it exactly the same is not really important.

At the end of the day, this is just bikeshedding about how to collapse ultra specific alerting levels into a few generic ones. E.g. RFC 5424 defines 8 separate log levels for syslog and, while that's not a ceiling by any means, it's easy to see how there's already not really going to be a universally agreed way to collapse even just these down to 4 categories.

Re: Log level 'error' should mean that something needs to be fixed

#43

Post by colechristensen » Sat, Dec 20, 2025, 5:25 PM UTC

mfuzzey wrote: post #8

I think it's difficult to say without knowing how the system is deployed and administered. "If a SMTP mailer trying to send email to somewhere logs 'cannot contact port 25 on ', that is not an error in the local system" Maybe or maybe not. If the connection problem is really due to the remote host then that's not the problem of the sender. But maybe the local network interface is down, maybe there's a local firewall…

How about this:

- An error is an event that someone should act on. Not necessarily you. But if it's not an event that ever needs the attention of a person then the severity is less than an error.

Examples: Invalid credentials. HTTP 404 - Not Found, HTTP 403 Forbidden, (all of the HTTP 400s, by definition)

It's not my problem as a site owner if one of my users entered the wrong URL or typed their password wrong, but it's somebody's problem.

A warning is something that A) a person would likely want to know and B) wouldn't necessarily need to act on

INFO is for something a person would likely want to know and unlikely needs action

DEBUG is for something likely to be helpful

TRACE is for just about anything that happens

EMERG/CRIT are for significant errors of immediate impact

PANIC the sky is falling, I hope you have good running shoes

Re: Log level 'error' should mean that something needs to be fixed

#44

Post by magicalhippo » Sat, Dec 20, 2025, 5:26 PM UTC

rwmj wrote: post #21

And the second rule is make all your error messages actionable . By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).

This can be difficult or just not possible.

What is possible is to include as much information about what the system was trying to do. If there's an file IO error, include the the full path name. Saying "file not found" without saying which file was not found infuriates me like few other things.

If some required configuration option is not defined, include the name of the configuration option and from where it tried to find said configuration (config files, environment, registry etc). And include the detailed error message from the underlying system if any.

Regular users won't have a clue how to deal with most errors anyway, but by including details at least someone with some system knowledge has a chance of figuring out how to fix or work around the issue.

Re: Log level 'error' should mean that something needs to be fixed

#45

Post by eterm » Sat, Dec 20, 2025, 5:30 PM UTC

How I'd personally like to treat them:

  - Critical / Fatal:  Unrecoverable without human intervention, someone needs to get out of bed, now.
  - Error : Recoverable without human intervention, but not without data / state loss. Must be fixed asap. An assumption didn't hold.
  - Warning: Recoverable without intervention. Must have an issue created and prioritised. ( If business as usual, this could be downgrading to INFO. )

The main difference therefore between error and warning is, "We didn't think this could happen" vs "We thought this might happen".

So for example, a failure to parse JSON might be an error if you're responsible for generating that serialisation, but might be a warning if you're not.

Re: Log level 'error' should mean that something needs to be fixed

#46

Post by pixl97 » Sat, Dec 20, 2025, 5:35 PM UTC

rwmj wrote: post #21

And the second rule is make all your error messages actionable . By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).

So what error do you put if the server is over 500 miles away?

https://web.mit.edu/jemorris/humor/500-miles

Or you can't connect because of a path MTU error.

Or because the TTL is set to low?

Your software at the server level has no idea what's going wrong at the network level, all you can send is some kind of network problem message.

Re: Log level 'error' should mean that something needs to be fixed

#47

Post by uniq7 » Sat, Dec 20, 2025, 5:37 PM UTC

chongli wrote: post #38

rwmj wrote: post #21

And the second rule is make all your error messages actionable . By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).

Suppose I'm writing an http server and the error is caused by a flaky power supply causing the disk to lose power when the server attempts to read a file that's been requested. How is the http server supposed to diagnose this or any other hardware fault? Furthermore, why should it even be the http server's responsibility to know about hardware issues at all?

The error doesn't need to be extremely specific or point to the actual root cause.

In your example,

- "Error while serving file" would be a bad error message,

- "Failed to read file 'foo/bar.html'" would be acceptable, and

- "Failed to read file 'foo/bar.html' due to EIO: Underlying device error (disk failure, I/O bus error). Please check the disk integrity." would be perfect (assuming the http server has access to the underlying error produced by the read operation).

Re: Log level 'error' should mean that something needs to be fixed

#48

Post by mewpmewp2 » Sat, Dec 20, 2025, 5:38 PM UTC

eterm wrote: post #45

How I'd personally like to treat them: - Critical / Fatal: Unrecoverable without human intervention, someone needs to get out of bed, now. - Error : Recoverable without human intervention, but not without data / state loss. Must be fixed asap. An assumption didn't hold. - Warning: Recoverable without intervention. Must have an issue created and prioritised. ( If business as usual, this could be downgrading to INFO. )…

What if you are integrated to a third party app and it gives you 5xx once? What do you log it as, and let's say after a retry it is fine.

Re: Log level 'error' should mean that something needs to be fixed

#49

Post by teo_zero » Sat, Dec 20, 2025, 5:41 PM UTC

This doesn't resonate with my experience. I place the line between a warning and an error whether the operation can or can't be completed.

A connection timed out, retrying in 30 secs? That's a warning. Gave up connecting after 5 failed attempts? Now that's an error.

I don't care so much if the origin of the error is within the program, or the system, or the network. If I can't get what I'm asking for, it can't be a mere warning.

Re: Log level 'error' should mean that something needs to be fixed

#50

Post by DanHulton » Sat, Dec 20, 2025, 5:41 PM UTC

colechristensen wrote: post #43

mfuzzey wrote: post #8

I think it's difficult to say without knowing how the system is deployed and administered. "If a SMTP mailer trying to send email to somewhere logs 'cannot contact port 25 on ', that is not an error in the local system" Maybe or maybe not. If the connection problem is really due to the remote host then that's not the problem of the sender. But maybe the local network interface is down, maybe there's a local firewall…

How about this: - An error is an event that someone should act on. Not necessarily you. But if it's not an event that ever needs the attention of a person then the severity is less than an error. Examples: Invalid credentials. HTTP 404 - Not Found, HTTP 403 Forbidden, (all of the HTTP 400s, by definition) It's not my problem as a site owner if one of my users entered the wrong URL or typed their password wrong, but i…

If you're logging and reporting on ERRORs for 400s, then your error triage log is going to be full of things like a user entering a password with insufficient complexity or trying to sign up with an email address that already exists in your system.

Some of these things can be ameliorated with well-behaved UI code, but a lot cannot, and if your primary product is the API, then you're just going to have scads of ERRORs to triage where there's literally nothing you can do.

I'd argue that anything that starts with a 4 is an INFO, and if you really wanted to be through, you could set up an alert on the frequency of these errors to help you identify if there's a broad problem.