Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log level 'error' should mean that something needs to be fixed

utcc.utoronto.ca

Post reply on HN
Print view View original

261–270 of 313 posts

Re: Log level 'error' should mean that something needs to be fixed

#261

Post by MereInterest »

lock1 wrote: post #236
usefulcat wrote: post #187

Earlier quoted context omitted.

Ideally a logging library should at least not make it easy to make that kind of mistake.

Ideally , but realistically, I have never heard of any major programming language that allows you to express "this function only accepts static constant string literal".

In Rust, this can almost be expressed as `arg: &'static str` to accept a reference to a string whose lifetime never ends. I say “almost” because this allows both string literals and references to static (but dynamically generated) string.

For Rust’s macros, a literal can be expressed as `$arg:lit`. This does allow other literals as well, such as int or float literals, but typically the generated code would only work for a string literal.

Re: Log level 'error' should mean that something needs to be fixed

#262

Post by heisenbit »

cwillu wrote: post #159
MobiusHorizons wrote: post #122

Earlier quoted context omitted.

What you are proposing sounds like a nightmare to debug. The high level perspective of the operation is of course valuable for determining if an investigation is necessary, but the low level perspective in the library code is almost always where the relevant details are hiding. Not logging these details means you are in the dark about anything your abstractions are hiding from higher level code (which is usually a lo…

Those details don't belong in the error log level, that's what info or trace is for.

Trace can become so voluminous that it is switched on only on a need basis which can be too late for rare events. Also trace level as more a need to use debug tool tends to be less scrutinized for exposing sensitive data making it unsuitable for continuous operation or use in live production.

Re: Log level 'error' should mean that something needs to be fixed

#263

Post by stefanfisk »

lock1 wrote: post #236
usefulcat wrote: post #187

Earlier quoted context omitted.

Ideally a logging library should at least not make it easy to make that kind of mistake.

Ideally , but realistically, I have never heard of any major programming language that allows you to express "this function only accepts static constant string literal".

Even PHP has that these days via static analysis https://phpstan.org/writing-php-code/phpdoc-types#other-adva...

Re: Log level 'error' should mean that something needs to be fixed

#264

Post by georgefrowny »

oofbey wrote: post #89
georgefrowny wrote: post #84

Easy to say, but there's "yes we know this is wrong but this will have to do for now" and "we don't expect to see this in real life unless something has gone sideways".

At scale the rare events start to happen reliably. Hardware failures almost certainly cause ERROR conditions. Network glitches. Our production system pages oncall for any errors. At night it will only wake somebody up for a whole bunch of errors. This discipline forces us to take a look at every ERROR and decide if it is spurious and out of our control or something we can deal with. At some point our production syste…

I think if someone is going be gotten out of bed that would be a critical rather then error. Generally I'd say in a large "live" system, errors end up raising Jira tickets, criticals end up ringing phones.

Re: Log level 'error' should mean that something needs to be fixed

#265

Post by IgorPartola »

p2detar wrote: post #259
eterm wrote: post #45

How I'd personally like to treat them: - Critical / Fatal: Unrecoverable without human intervention, someone needs to get out of bed, now. - Error : Recoverable without human intervention, but not without data / state loss. Must be fixed asap. An assumption didn't hold. - Warning: Recoverable without intervention. Must have an issue created and prioritised. ( If business as usual, this could be downgrading to INFO. )…

Yea but instead of log Critical/Fatal and go on, I would just panic() the program. To the other definitions I agree - everything else is recoverable, because the program still runs. Warning to me is an error that has very little business logic side effects/impact as opposed to an Error, but still requires attention.

I write a lot of backend web code that often talks to external services. So for example the user wants to add a shipping address to their profile but the address verification API responds with a 500. That is an expected error: sometimes it can happen. I want to log it but I do not want a trace back or anything like that.

On the other hand it could be that the API had changed slightly. Say they for some reason decided to rename the input parameter postcode to postal_code and I didn’t change my code to fix this. This is 100% a programming error that would be classified as critical but I would not want to panic() the entire server process over it. I just want an alert that hey there is a programming error, go fix it.

But what could also happen is that when I try to construct a request for the external API and the OS is out of memory. Then I want to just crash the process and rely on automatic process restarts to bring it back up. BTW logging an error after malloc() returns NULL needs to be done carefully since you cannot allocate more memory for things like a new log string.

Re: Log level 'error' should mean that something needs to be fixed

#266

Post by IgorPartola »

srdjanr wrote: post #237
cpburns2009 wrote: post #72

Earlier quoted context omitted.

It really depends on the third party service. For service A, a 500 error may be common and you just need to try again, and a descriptive 400 error indicates the original request was actually handled. In these cases I'd log as a warning. For service B, a 500 error may indicate the whole API is down, in which case I'd log a warning and not try any more requests for 5 minutes. For service C, a 500 error may be an anomal…

What's the difference between B and C? API being down seems like an anomaly. Also, you can't know how frequently you'll get 500s at the time you're doing integration, so you'll have to go back after some time to revisit log severities. Which doesn't sound optimal.

Exactly. What’s worse is that if you have something like a web service that calls an external API, when that API goes down your log is going to be littered with errors and possibly even tracebacks which is just noise. If you set up a simple “email me on error” kind of service you will get as many emails as there were user requests.

In theory some sort of internal API status tracking thing would be better that has some heuristic of is the API up or down and the error rate. It should warn you when the API is down and when it comes back up. Logging could still show an error or a warning for each request but you don’t need to get an email about each one.

Re: Log level 'error' should mean that something needs to be fixed

#267

Post by [deleted] »

NewJazz wrote: post #234
demurgos wrote: post #224

Earlier quoted context omitted.

I feel like there's a parallel with SQL where you want to discourage manual interpolation. Taking inspiration from it may help: you may not fully solve it but there are some API ideas and patterns. A logging framework may have the equivalent of prepared statements. You may also nudge usage where the raw string API is `log.traceRaw(String rawMessage)` while the parametrized one has the nicer naming `log.trace(Template…

You can have 0 parameters and the template is a string...

[deleted]

Re: Log level 'error' should mean that something needs to be fixed

#268

Post by bluGill »

demurgos wrote: post #245
NewJazz wrote: post #234

Earlier quoted context omitted.

You can have 0 parameters and the template is a string...

The point of my message is that you should avoid the `log(string)` signature. Even if it's appealing, it's an easy perf trap. There are many ideas if you look at SQL libs. In my example I used a different type but there other solutions. Be creative. logger.log(new Template("foo"))` logger.log("foo", []) logger.prepare("foo").log()

Unless log() is a macro of some sort that expands to if(logEnabled){internalLog(string)} - which a good optimizer will see through and not expand the string when logging is disabled.

Re: Log level 'error' should mean that something needs to be fixed

#269

Post by bluGill »

kgklxksnrb wrote: post #255
TZubiri wrote: post #142

Earlier quoted context omitted.

We call those warnings, and it's very common to downgrade errors to warnings by wrapping an exception and printing the trace as you would an exception.

Logging warnings are cowardly, you just push the decision to the log consumer to decide if the error should be acted on. Warnings are just errors that no one wants to deal with.

Warnings are for where you expect someplace else to know/log if it really is an error but it might also be normal. You might log why a file io operation failed: if the caller recovers somehow it isn't an errer, but if they can't they log an error and when investigating the warning gives the detail you need to figure it out.

Re: Log level 'error' should mean that something needs to be fixed

#270

Post by t43562 »

Errors can be recovered automatically sometimes but at the level at which you log them you don't know if that's going to happen. I therefore think this suggestion is not easy to follow.

Even if your libraries use nothing but exceptions or return codes you still end up with levels. You still end up with logs that have information in them that gets ignored when it shouldn't be because there's so much noise that people get tired of all the "cries of wolf."

Occasionally one is at a high enough level to know for sure that something needs fixing and for this I use "CRITICAL" which is my code for "absolutely sure that you can't ignore this."

IMO it's about time AI was looking at the logs to find out if there was something we really need to be alerted to.

Post reply on HN