Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log level 'error' should mean that something needs to be fixed

utcc.utoronto.ca

Post reply on HN
Print view View original

71–80 of 313 posts

Re: Log level 'error' should mean that something needs to be fixed

#71

Post by wredcoll »

sysguest wrote: post #55
eterm wrote: post #45

How I'd personally like to treat them: - Critical / Fatal: Unrecoverable without human intervention, someone needs to get out of bed, now. - Error : Recoverable without human intervention, but not without data / state loss. Must be fixed asap. An assumption didn't hold. - Warning: Recoverable without intervention. Must have an issue created and prioritised. ( If business as usual, this could be downgrading to INFO. )…

hmm maybe we need extra representation? eg: 2.0 for "trace" / 1.0 for "debug" / 0.0 for "info" / -1.0 for "warn" / -2.0 for "error that can be handled"

I said this elsewhere, but the point here is what the humans involved are supposed to do with this info. Do I literally get out of bed on an error log or do I grep for them once or twice a month?

Re: Log level 'error' should mean that something needs to be fixed

#72

Post by cpburns2009 »

mewpmewp2 wrote: post #48
eterm wrote: post #45

How I'd personally like to treat them: - Critical / Fatal: Unrecoverable without human intervention, someone needs to get out of bed, now. - Error : Recoverable without human intervention, but not without data / state loss. Must be fixed asap. An assumption didn't hold. - Warning: Recoverable without intervention. Must have an issue created and prioritised. ( If business as usual, this could be downgrading to INFO. )…

What if you are integrated to a third party app and it gives you 5xx once? What do you log it as, and let's say after a retry it is fine.

It really depends on the third party service.

For service A, a 500 error may be common and you just need to try again, and a descriptive 400 error indicates the original request was actually handled. In these cases I'd log as a warning.

For service B, a 500 error may indicate the whole API is down, in which case I'd log a warning and not try any more requests for 5 minutes.

For service C, a 500 error may be an anomaly and treat it as hard error and log as error.

Re: Log level 'error' should mean that something needs to be fixed

#73

Post by mewpmewp2 »

wredcoll wrote: post #68
mewpmewp2 wrote: post #61

Earlier quoted context omitted.

You are not the OP, but I think I was trying to point out this example case in relation to their descriptions of Error/Warnings. This scenario may or may not yield in data/state loss, it may also be something that you, yourself can't immediately fix. And if it's temporary, what is the point of creating an issue and prioritizing. I guess my point is that to any such categorization of errors or warnings there are way t…

Like so many things in this industry the point is establishing a shared meaning for all the humans involved, regardless of how uninvolved people think. That being said, I find tying the level to expected action a more useful way to classify them.

But what I also see frequently is people trying to do the impossible and idealistic things because they read somewhere that something should mean X, when things are never so clearly cut, so either it is not such a simplistic issue and should be understood as not such a simple issue, or there might be a better more practical definition for it. We should first start from what are we using Logs for. Are we using those for debugging, or so we get alerted or both?

If for debugging, the levels seem relevant in the sense of how quickly we are able to use that information to understand what is going wrong. Out of potential sea of logs we want to see first what were the most likely culprits for something causing something to go wrong. So the higher the log level, the higher likelihood of this event causing something to go wrong.

If for alerting, they should reflect on how bad is this particular thing happening for the business and would help us to set a threshold for when we page or have to react to something.

Re: Log level 'error' should mean that something needs to be fixed

#74

Post by raldi »

aqme28 wrote: post #65

I agree with this take in a steady state, but the process of building software is just that-- it's a process. So it's natural for error messages to be expected, as you progressively add and then clear up edge cases.

Exactly: When you're building software, it has lots of defects (and, thus, error logging). When it's mature, it should have few defects, and thus few error logs, and each one that remains is a bug that should be fixed.

Re: Log level 'error' should mean that something needs to be fixed

#75

Post by raldi »

ImPostingOnHN wrote: post #32
raldi wrote: post #23

Earlier quoted context omitted.

You have an alert on what users actually care about, like the overall success rate. When it goes off, you check the WARNING log and metric dashboard and see that requests are timing out.

That is a lagging indicator. By the time you're alerted, you've already failed by letting users experience an issue.

What alternative would you propose? Page the oncall whenever there's a single query timeout?

Re: Log level 'error' should mean that something needs to be fixed

#76

Post by layer8 »

> When implementing logging, it's important to distinguish between an error from the perspective of an individual operation and an error from the perspective of the overall program or system. Individual operations may well experience errors that are not error level log events for the overall program. You could say that an operation error is anything that prevents an operation from completing successfully, while a program level error is something that prevents the program as a whole from working right.

This is a nontrivial problem when using properly modularized code and libraries that perform logging. They can’t tell whether their operational error is also a program-level error, which can depend on usage context, but they still want to log the operational error themselves, in order to provide the details that aren’t accessible to higher-level code. This lower-level logging has to choose some status.

Should only “top-level” code ever log an error? That can make it difficult to identify the low-level root causes of a top-level failure. It also can hamper modularization, because it means you can’t repackage one program’s high-level code as a library for use by other programs, without somehow factoring out the logging code again.

Re: Log level 'error' should mean that something needs to be fixed

#78

Post by ivan_gammel »

layer8 wrote: post #76

> When implementing logging, it's important to distinguish between an error from the perspective of an individual operation and an error from the perspective of the overall program or system. Individual operations may well experience errors that are not error level log events for the overall program. You could say that an operation error is anything that prevents an operation from completing successfully, while a pro…

Libraries should not log on levels above DEBUG, period. If there’s something worthy for reporting on higher levels, pass this information to client code, either as an event, or as an exception or error code.

Re: Log level 'error' should mean that something needs to be fixed

#79

Post by rwmj »

1123581321 wrote: post #39
rwmj wrote: post #21

And the second rule is make all your error messages actionable . By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).

Can you please explain this? That sounds like identifying bugs but not fixing them but I realize you don’t mean that. One hopes the context information in the error will make it actionable when it occurs, never completely successfully, of course.

Here's an example of a bug that I filed about non-actionable error messages: https://github.com/karmab/kcli/issues/456

The first error message was "No usable public key found, which is required for the deployment" which doesn't tell me what I have to do to correct the problem. Nothing about even where it's looking for keys, what is supposed to create the key or how I am supposed to create the key.

There are other examples and discussion of what they should say in the link.

Edit: Here's another one that I filed: https://github.com/containers/podman/issues/20775

Re: Log level 'error' should mean that something needs to be fixed

#80

Post by RaftPeople »

eterm wrote: post #45

How I'd personally like to treat them: - Critical / Fatal: Unrecoverable without human intervention, someone needs to get out of bed, now. - Error : Recoverable without human intervention, but not without data / state loss. Must be fixed asap. An assumption didn't hold. - Warning: Recoverable without intervention. Must have an issue created and prioritised. ( If business as usual, this could be downgrading to INFO. )…

> The main difference therefore between error and warning is, "We didn't think this could happen" vs "We thought this might happen".

What about conditions like "we absolutely knew this would happen regularly, but it's something that prevents the completion of the entire process which is absolutely critical to the organization"

The notion of an "error" is very context dependent. We usually use it to mean "can not proceed with action that is required for the successful completion of this task"

Post reply on HN