Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log level 'error' should mean that something needs to be fixed

utcc.utoronto.ca

Post reply on HN
Print view View original

101–110 of 313 posts

Re: Log level 'error' should mean that something needs to be fixed

#101

Post by 1718627440 »

ivan_gammel wrote: post #78
layer8 wrote: post #76

> When implementing logging, it's important to distinguish between an error from the perspective of an individual operation and an error from the perspective of the overall program or system. Individual operations may well experience errors that are not error level log events for the overall program. You could say that an operation error is anything that prevents an operation from completing successfully, while a pro…

Libraries should not log on levels above DEBUG, period. If there’s something worthy for reporting on higher levels, pass this information to client code, either as an event, or as an exception or error code.

Why? Whats wrong with logging it and passing the log object to the caller? The caller can still modify the log entry however it pleases?

Re: Log level 'error' should mean that something needs to be fixed

#102

Post by layer8 »

ivan_gammel wrote: post #86
layer8 wrote: post #83

Earlier quoted context omitted.

From a code modularization point of view, there shouldn’t really be much of a difference between programs and libraries. A program is just a library with a different calling convention. I like to structure programs such that their actual functionality could be reused as a library in another program. This is difficult to reconcile with libraries only logging on a debug level.

The main difference is that library is not aware of the context of the execution of the code, so cannot decide, whether the problem is expected, recoverable or severe.

The same is true for programs that are being invoked. The program only knows relative to its own purpose, and the same is again true for libraries. I don’t see the difference, other than, as already mentioned, the mechanism of program vs. library invocation.

Consider a Smalltalk-like system, or something like TCL, that doesn’t distinguish between programs and libraries regarding invocation mechanism. How would you handle logging in that case?

Re: Log level 'error' should mean that something needs to be fixed

#103

Post by 1718627440 »

notatoad wrote: post #63
AndroTux wrote: post #52

“cannot contact port 25 on ” may very well be a configuration error. How should the program know?

>How should the program know? if we're talking about logs from our own applications that we have written, the program should know because we can write it in a way that it knows. user-defined config should be verified before it is used. make a ping to port 25 to see if it works before you start using that config for actual operation. if it fails the verification step, that's not an error that needs to be logged.

So when the random error on a remote party happens at one time your system ignores it, bu when it happens at another time, it prevents the server from booting? That's a very brittle system.

Re: Log level 'error' should mean that something needs to be fixed

#104

Post by ivan_gammel »

1718627440 wrote: post #101
ivan_gammel wrote: post #78

Earlier quoted context omitted.

Libraries should not log on levels above DEBUG, period. If there’s something worthy for reporting on higher levels, pass this information to client code, either as an event, or as an exception or error code.

Why? Whats wrong with logging it and passing the log object to the caller? The caller can still modify the log entry however it pleases?

Practicality. It is excessive for client code to calibrate library logging level. It’s ok to do it in logging configuration, but having an entry for every library there is also excessive. It is reasonable to expect that dev/staging may have base level at DEBUG and production will have base level at INFO, so that a library following the convention will not require extra effort to prevent log spam in production. Yes, we have entire logging industry around aggregation of terabytes of logs, with associated costs, but do you really need that? In other words, are we developers too lazy to adapt the sane logging policy, which actually requires minimum effort, and will just burn the company money for nothing?

Re: Log level 'error' should mean that something needs to be fixed

#105

Post by tcpkump »

notatoad wrote: post #63
AndroTux wrote: post #52

“cannot contact port 25 on ” may very well be a configuration error. How should the program know?

>How should the program know? if we're talking about logs from our own applications that we have written, the program should know because we can write it in a way that it knows. user-defined config should be verified before it is used. make a ping to port 25 to see if it works before you start using that config for actual operation. if it fails the verification step, that's not an error that needs to be logged.

What about when the mail server endpoint has changed, and for whatever reason, this configuration wasn’t updated? This is a common scenario when dealing with legacy infrastructure in my experience.

Re: Log level 'error' should mean that something needs to be fixed

#107

Post by Hizonner »

raldi wrote: post #10
Hizonner wrote: post #9

Earlier quoted context omitted.

> Database timeout (the database is owned by a separate oncall rotation that has alerts when this happens) So people writing software are supposed to guess how your organization assigns responsibilities internally? And you're sure that the database timeout always happens because there's something wrong with the database, and never because something is wrong on your end?

No; I’m not understanding your point about guessing. Could you restate? As for queries that time out, that should definitely be a metric, but not pollute the error loglevel, especially if it’s something that happens at some noisy rate all the time.

> No; I’m not understanding your point about guessing. Could you restate?

In the general case, the person writing the software has no way of knowing that "the database is owned by a separate oncall rotation". That's about your organization chart.

Admittedly, they'd be justified in assuming that somebody is paying attention to the database. On the other hand, they really can't be sure that the database is going to report anything useful to anybody at all, or whether it's going to report the salient details. The database may not even know that the request was ever made. Maybe the requests are timing out because they never got there. And definitely maybe the requests are timing out because you're sending too many of them.

I mean, no, it doesn't make sense to log a million identical messages, but that's rate limiting. It's still an error if you can't access your database, and for all you know it's an error that your admin will have to fix.

As for metrics, I tend to see those as downstream of logs. You compute the metric by counting the log messages. And a metric can't say "this particular query failed". The ideal "database timeout" message should give the exact operation that timed out.

Re: Log level 'error' should mean that something needs to be fixed

#108

Post by masswerk »

eterm wrote: post #45

How I'd personally like to treat them: - Critical / Fatal: Unrecoverable without human intervention, someone needs to get out of bed, now. - Error : Recoverable without human intervention, but not without data / state loss. Must be fixed asap. An assumption didn't hold. - Warning: Recoverable without intervention. Must have an issue created and prioritised. ( If business as usual, this could be downgrading to INFO. )…

Also, warnings for ambiguous results.

For example, when a process implies a conversion according to the contract/convention, but we know that this conversion may be not the expected result and the input may be based on semantic misconceptions. E.g., assemblers and contextually truncated values for operands: while there's no issue with the grammar or syntax or intrinsic semantics, a higher level misconception may be involved (e.g., regarding address modes), resulting in a correct but still non-functional output. So, "In this individual case, there may be or may be not an issue. Please, check. (Not resolvable on our end.)"

(Disclaimer: I know that this is a very much classic computing and that this is now mostly moved to the global TOS, but still, it's the classic example for a warning.)

Re: Log level 'error' should mean that something needs to be fixed

#109

Post by danaris »

ImPostingOnHN wrote: post #32
raldi wrote: post #23

Earlier quoted context omitted.

You have an alert on what users actually care about, like the overall success rate. When it goes off, you check the WARNING log and metric dashboard and see that requests are timing out.

That is a lagging indicator. By the time you're alerted, you've already failed by letting users experience an issue.

Well, yes. If the cable falls out of the server (or there's a power outage, or a major DDoS attack, or whatever), your users are going to experience that before you are aware of it. Especially if it's in the middle of the night and you don't have an active night shift.

Expecting arbitrary services to be able to deal with absolutely any kind of failure in such a way that users never notice is deeply unrealistic.

Re: Log level 'error' should mean that something needs to be fixed

#110

Post by schrodinger »

layer8 wrote: post #83
ivan_gammel wrote: post #78

Earlier quoted context omitted.

Libraries should not log on levels above DEBUG, period. If there’s something worthy for reporting on higher levels, pass this information to client code, either as an event, or as an exception or error code.

From a code modularization point of view, there shouldn’t really be much of a difference between programs and libraries. A program is just a library with a different calling convention. I like to structure programs such that their actual functionality could be reused as a library in another program. This is difficult to reconcile with libraries only logging on a debug level.

I see your point, but disagree on a practical level. Libraries are being used while you’re in “developer” mode, while programs are used in “user” mode (trying awkwardly to differentiate between _being_ a developer and currently developing code around that library.

Usually a program is being used by the user to accomplish something, and if logging is meaningful than either in a cli context or a server context. In both cases, errors are more often being seen by people/users than by code. Therefore printing them to logs make sense.

While a lib is being used by a program. So it has a better way to communicate problems with the caller (and exceptions, error values, choose the poison of your language). But I almost never want a library to start logging shit because it’s almost guaranteed to not follow the same conventions as I do in my program elsewhere. Return me the error and let me handle.

It’s analogous to how Go has an implicit rule of that a library should never let a panic occur outside the library. Internally, fine. But at the package boundary, you should catch panics and return them as an error. You don’t know if the caller wants the app to die because it an error in your lib!

Post reply on HN