Log level 'error' should mean that something needs to be fixed
utcc.utoronto.ca
Log level 'error' should mean that something needs to be fixed
1–10 of 313 posts
Re: Log level 'error' should mean that something needs to be fixed
#2This is the standard I use as well. In general, my rule of thumb is that if something is logging error, it would have been perfectly reasonable for the program to respond by crashing, and the only reason it didn't is that it's executing in some kind of larger context that wants to stay up in the event of the failure of an individual component (like one handler suffering a query that hangs it and having to be terminated by its monitoring program in a program with multiple threads serving web requests). In contrast, something like an ill-formed web query from an untrusted source isn't even an error because you can't force untrusted sources to send you correctly formed input.
Warning, in contrast, is what I use for a condition that the developer predicted and handled but probably indicates the larger context is bad, like "this query arrived from a trusted source but had a configuration so invalid we had to drop it on the floor, or we assumed a default that allowed us to resolve the query but that was a massive assumption and you really should change the source data to be explicit." Warning is also where I put things like "a trusted source is calling a deprecated API, and the deprecation notification has been up long enough that they really should know better by now."
Where all of this matters is process. Errors trigger pages. Warnings get bundled up into a daily report that on-call is responsible for following up on, sometimes by filing tickets to correct trusted sources and sometimes by reaching out to owners of trusted sources and saying "Hey, let's synchronize on your team's plan to stop using that API we declared is going away 9 months ago."
Re: Log level 'error' should mean that something needs to be fixed
#3Yes. Examples of non-defects that should not be in the ERROR loglevel:
* Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens)
* ISE in downstream service (return HTTP 5xx and increment a metric but don’t emit an error log)
* Network error
* Downstream service overloaded
* Invalid request
Basically, when you make a request to another service and get back a status code, your handler should look like:
logfunc = logger.error if 400 Re: Log level 'error' should mean that something needs to be fixed
#4Errors mean I get alerted. Zero tolerance on that from my side.
Re: Log level 'error' should mean that something needs to be fixed
#5This is the standard I use as well. In general, my rule of thumb is that if something is logging error, it would have been perfectly reasonable for the program to respond by crashing, and the only reason it didn't is that it's executing in some kind of larger context that wants to stay up in the event of the failure of an individual component (like one handler suffering a query that hangs it and having to be terminat…
Right. If staging or the canary is logging errors, you block/abort the deploy. If it’s logging warnings, that’s normal.
Re: Log level 'error' should mean that something needs to be fixed
#6Yes. Examples of non-defects that should not be in the ERROR loglevel: * Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens) * ISE in downstream service (return HTTP 5xx and increment a metric but don’t emit an error log) * Network error * Downstream service overloaded * Invalid request Basically, when you make a request to another service and get back a status cod…
4xx is for invalid requests. You wouldn't log a 404 as an error
Re: Log level 'error' should mean that something needs to be fixed
#7Yes. Examples of non-defects that should not be in the ERROR loglevel: * Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens) * ISE in downstream service (return HTTP 5xx and increment a metric but don’t emit an error log) * Network error * Downstream service overloaded * Invalid request Basically, when you make a request to another service and get back a status cod…
4xx is for invalid requests. You wouldn't log a 404 as an error
I’m talking about codes you receive from services you call out to.
Re: Log level 'error' should mean that something needs to be fixed
#8I think it's difficult to say without knowing how the system is deployed and administered.
"If a SMTP mailer trying to send email to somewhere logs 'cannot contact port 25 on ', that is not an error in the local system"
Maybe or maybe not. If the connection problem is really due to the remote host then that's not the problem of the sender. But maybe the local network interface is down, maybe there's a local firewall rule blocking it,...
If you know the deployment scenario then you can make reasonable decisions on logging levels but quite often code is generic and can be deployed in multiple configurations so that's hard to do
Re: Log level 'error' should mean that something needs to be fixed
#9Yes. Examples of non-defects that should not be in the ERROR loglevel: * Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens) * ISE in downstream service (return HTTP 5xx and increment a metric but don’t emit an error log) * Network error * Downstream service overloaded * Invalid request Basically, when you make a request to another service and get back a status cod…
> Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens)
So people writing software are supposed to guess how your organization assigns responsibilities internally? And you're sure that the database timeout always happens because there's something wrong with the database, and never because something is wrong on your end?
Re: Log level 'error' should mean that something needs to be fixed
#10Yes. Examples of non-defects that should not be in the ERROR loglevel: * Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens) * ISE in downstream service (return HTTP 5xx and increment a metric but don’t emit an error log) * Network error * Downstream service overloaded * Invalid request Basically, when you make a request to another service and get back a status cod…
> Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens) So people writing software are supposed to guess how your organization assigns responsibilities internally? And you're sure that the database timeout always happens because there's something wrong with the database, and never because something is wrong on your end?
No; I’m not understanding your point about guessing. Could you restate?
As for queries that time out, that should definitely be a metric, but not pollute the error loglevel, especially if it’s something that happens at some noisy rate all the time.