Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log level 'error' should mean that something needs to be fixed

utcc.utoronto.ca

Post reply on HN
Print view View original

31–40 of 313 posts

Re: Log level 'error' should mean that something needs to be fixed

#31

Post by quectophoton »

jayofdoom wrote: post #24

In OpenStack, we explicitly document what our log levels mean; I think this is valuable from both an Operator and Developer perspective. If you're a new developer, without a sense of what log levels are for, it's very prescriptive and helpful. For an operator, it sets expectations. https://docs.openstack.org/oslo.log/latest/user/guidelines.h... FWIW, "ERROR: An error has occurred and an administrator should research…

Thank you, this (and jillesvangurp's comment) sounds way more reasonable than the article's suggestion.

If I have a daily cron job that is copying files to a remote location (e.g. backups), and the _operation_ fails because for some reason the destination is not writable.

Your suggestion would get me _both_ alerts, as I want; the article's suggestion would not alert me about the operation failing because, after all, it's not something happening in the local system, the local program is well configured, and it's "working as expected" because it doesn't need neither code nor configuration fixing.

Re: Log level 'error' should mean that something needs to be fixed

#32

Post by ImPostingOnHN »

raldi wrote: post #23
dnautics wrote: post #19

let's say you a bunch of database timeouts in a row. this might mean that nothing needs to be fixed. But also, the "thing that needs to be fixed" might be "the ethernet cable fell out the back of your server". How do you know?

You have an alert on what users actually care about, like the overall success rate. When it goes off, you check the WARNING log and metric dashboard and see that requests are timing out.

That is a lagging indicator. By the time you're alerted, you've already failed by letting users experience an issue.

Re: Log level 'error' should mean that something needs to be fixed

#33

Post by nlawalker »

shadowgovt wrote: post #2

This is the standard I use as well. In general, my rule of thumb is that if something is logging error, it would have been perfectly reasonable for the program to respond by crashing, and the only reason it didn't is that it's executing in some kind of larger context that wants to stay up in the event of the failure of an individual component (like one handler suffering a query that hangs it and having to be terminat…

It seems that the easier rule of thumb, then, is that "application logic should never log an error on its own behalf unless it terminates immediately after", and that error-level log entries should only ever be generated from a higher-level context by something else that's monitoring for problems that the application code itself didn't anticipate.

Re: Log level 'error' should mean that something needs to be fixed

#34

Post by hyperadvanced »

rwmj wrote: post #21

And the second rule is make all your error messages actionable . By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).

This is just plain wrong, I vehemently disagree. What happens if a payment fails on my API, and today that means I need to go through a 20-step process with this pay provider, my database, etc. to correct that. But what’s worse is if this error happens 11,000 times and I run a script to do my 20 step process 11,000 times, but it turns out the error was raised in error. Additionally, because the error was so explicit about how to fix it, I didn’t talk to anyone. And of course, the suggested fix was out of date because docs lag vs. production software. Now I have 11,000 pissed off customers because I was trying to be helpful.

Re: Log level 'error' should mean that something needs to be fixed

#35

Post by throw3e98 »

rwmj wrote: post #21

And the second rule is make all your error messages actionable . By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).

Maybe that makes sense for a single-machine application where you also control the hardware. But for a networked/distributed system, or software that runs on the user's hardware, the action might involve a decision tree, and a log line is a poor way to convey that. We use instrumentation, alerting and runbooks for that instead, with the runbooks linking into a hyperlinked set of articles.

My 3D printer will try to walk you through basic fixes with pictures on the device's LCD panel, but for some errors it will display a QR code to their wiki which goes into a technical troubleshooting guide with complex instructions and tutorial videos.

Re: Log level 'error' should mean that something needs to be fixed

#36

Post by dragonwriter »

hnbear wrote: post #16

I have been particularly irritated in the past where people use a lower log level and include the higher log level string in the message, especially where it's then parsed, filtered, and alerted on my monitoring. eg. log level WARN, message "This error is...", but it then trips an error in monitoring and pages out. Probably breaching multiple rules here around not parsing logs like that, etc. But it's cropped up so m…

> I have been particularly irritated in the past where people use a lower log level and include the higher log level string in the message, especially where it's then parsed, filtered, and alerted on my monitoring.

If your parsing, filtering, and monitoring setup parses strings that happen to correspond to log level names in positions other than that of log levels as having the semantics of log levels, then that's a parsing/filtering error, not a logging error.

Re: Log level 'error' should mean that something needs to be fixed

#37

Post by electroly »

raldi wrote: post #10
Hizonner wrote: post #9

Earlier quoted context omitted.

> Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens) So people writing software are supposed to guess how your organization assigns responsibilities internally? And you're sure that the database timeout always happens because there's something wrong with the database, and never because something is wrong on your end?

No; I’m not understanding your point about guessing. Could you restate? As for queries that time out, that should definitely be a metric, but not pollute the error loglevel, especially if it’s something that happens at some noisy rate all the time.

I think OP is making two separate but related points, a general point and a specific point. Both involve guessing something that the error-handling code, on the spot, might not know.

1. When I personally see database timeouts at work it's rarely the database's fault, 99 times out of 100 it's the caller's fault for their crappy query; they should have looked at the query plan before deploying it. How is the error-handling code supposed to know? I log timeouts (that still fail after retry) as errors so someone looks at it and we get a stack trace leading me to the bad query. The database itself tracks timeout metrics but the log is much more immediately useful: it takes me straight to the scene of the crime. I think this is OP's primary point: in some cases, investigation is required to determine whether it's your service's fault or not, and the error-handling code doesn't have the information to know that.

2. As with exceptions vs. return values in code, the low-level code often doesn't know how the higher-level caller will classify a particular error. A low-level error may or may not be a high-level error; the low-level code can't know that, but the low-level code is the one doing the logging. The low-level logging might even be a third party library. This is particularly tricky when code reuse enters the picture: the same error might be "page the on-call immediately" level for one consumer, but "ignore, this is expected" for another consumer.

I think the more general point (that you should avoid logging errors for things that aren't your service's fault) stands. It's just tricky in some cases.

Re: Log level 'error' should mean that something needs to be fixed

#38

Post by chongli »

rwmj wrote: post #21

And the second rule is make all your error messages actionable . By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).

Suppose I'm writing an http server and the error is caused by a flaky power supply causing the disk to lose power when the server attempts to read a file that's been requested. How is the http server supposed to diagnose this or any other hardware fault? Furthermore, why should it even be the http server's responsibility to know about hardware issues at all?

Re: Log level 'error' should mean that something needs to be fixed

#39

Post by 1123581321 »

rwmj wrote: post #21

And the second rule is make all your error messages actionable . By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).

Can you please explain this? That sounds like identifying bugs but not fixing them but I realize you don’t mean that. One hopes the context information in the error will make it actionable when it occurs, never completely successfully, of course.

Re: Log level 'error' should mean that something needs to be fixed

#40

Post by morkalork »

andoando wrote: post #27
rwmj wrote: post #21

And the second rule is make all your error messages actionable . By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).

Error: Possible race condition, rewrite codebase

I have written out-of-band sanity checks that have caught race conditions, the recommendation is more like " that should be locked, isn't. Check what was merged and deployed in the last 24h, someone ducked it up"
Post reply on HN