And the second rule is make all your error messages actionable. By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).
Log level 'error' should mean that something needs to be fixed
21–30 of 313 posts
Re: Log level 'error' should mean that something needs to be fixed
#22I have been particularly irritated in the past where people use a lower log level and include the higher log level string in the message, especially where it's then parsed, filtered, and alerted on my monitoring. eg. log level WARN, message "This error is...", but it then trips an error in monitoring and pages out. Probably breaching multiple rules here around not parsing logs like that, etc. But it's cropped up so m…
Stuff like that is a good argument for using structured logging, but even if you are just parsing text logs, surely you can make the parser be a bit more specific when retrieving the log level.
Re: Log level 'error' should mean that something needs to be fixed
#23let's say you a bunch of database timeouts in a row. this might mean that nothing needs to be fixed. But also, the "thing that needs to be fixed" might be "the ethernet cable fell out the back of your server". How do you know?
You have an alert on what users actually care about, like the overall success rate. When it goes off, you check the WARNING log and metric dashboard and see that requests are timing out.
Re: Log level 'error' should mean that something needs to be fixed
#24In OpenStack, we explicitly document what our log levels mean; I think this is valuable from both an Operator and Developer perspective. If you're a new developer, without a sense of what log levels are for, it's very prescriptive and helpful. For an operator, it sets expectations.
https://docs.openstack.org/oslo.log/latest/user/guidelines.h...
FWIW, "ERROR: An error has occurred and an administrator should research the event." (vs WARNING: Indicates that there might be a systemic issue; potential predictive failure notice.)
Re: Log level 'error' should mean that something needs to be fixed
#25Earlier quoted context omitted.
> Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens) So people writing software are supposed to guess how your organization assigns responsibilities internally? And you're sure that the database timeout always happens because there's something wrong with the database, and never because something is wrong on your end?
No; I’m not understanding your point about guessing. Could you restate? As for queries that time out, that should definitely be a metric, but not pollute the error loglevel, especially if it’s something that happens at some noisy rate all the time.
> the database is owned by a separate oncall rotation
Not OP, but this part hits the same for me.
In the case your client app is killing the DB through too many calls (e.g. your cache is not working) you should be able to detect it and react, without waiting for the DB team to come to you after they investigated the whole thing.
But you can't know in advance if the DB connection errors are your fault or not, so logging it to cover the worse case scenario (you're the cause) is sensible.
Re: Log level 'error' should mean that something needs to be fixed
#26Earlier quoted context omitted.
No; I’m not understanding your point about guessing. Could you restate? As for queries that time out, that should definitely be a metric, but not pollute the error loglevel, especially if it’s something that happens at some noisy rate all the time.
> the database is owned by a separate oncall rotation Not OP, but this part hits the same for me. In the case your client app is killing the DB through too many calls (e.g. your cache is not working) you should be able to detect it and react, without waiting for the DB team to come to you after they investigated the whole thing. But you can't know in advance if the DB connection errors are your fault or not, so loggi…
I agree that you should detect this, just through a metric rather than putting DB timeouts in the ERROR loglevel.
Re: Log level 'error' should mean that something needs to be fixed
#27And the second rule is make all your error messages actionable . By that I mean it should tell me what action to take to fix the error (even if that action means hard work, tell me what I have to do).
Error: Possible race condition, rewrite codebase
Re: Log level 'error' should mean that something needs to be fixed
#28I think it's difficult to say without knowing how the system is deployed and administered. "If a SMTP mailer trying to send email to somewhere logs 'cannot contact port 25 on ', that is not an error in the local system" Maybe or maybe not. If the connection problem is really due to the remote host then that's not the problem of the sender. But maybe the local network interface is down, maybe there's a local firewall…
The point is that if your program itself take note of the error from the library it is ok. You, as the program owner, can decide what to do with it (error log or not).
But if you are the SMTP library and that you unilaterally log that as an error. That is an issue.
Re: Log level 'error' should mean that something needs to be fixed
#29I agree with the sentiment, although not sure if "error" is the right category/verbiage for actionable logs.
In an ideal world things like logs and alarms (alerting product support staff) should certainly cleanly separate things that are just informative, useful for the developer, and things that require some human intervention.
If you don't do this then it's like "the boy that cried wolf", and people will learn to ignore errors and alarms since you've trained them to understand that usually no action is needed. It's also useful to be able to grep though log files and distinguish failures of different categories, not just grep for specific failures.
Re: Log level 'error' should mean that something needs to be fixed
#30I just started playing in the Erlang ecosystem and they have EIGHT levels of logging messages. it seems crazily over-specific, but they are the champions of robust systems. I could live with 4 Error - alert me now. Warning - examine these later, Info - important context for investigations. Debug - usually off in prod.
The eight levels in Erlang are inherited from syslog, rather than something specific to Erlang itself.