Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log level 'error' should mean that something needs to be fixed

utcc.utoronto.ca

Post reply on HN
Print view View original

201–210 of 313 posts

Re: Log level 'error' should mean that something needs to be fixed

#201

Post by cyphar »

Too wrote: post #92
layer8 wrote: post #76

> When implementing logging, it's important to distinguish between an error from the perspective of an individual operation and an error from the perspective of the overall program or system. Individual operations may well experience errors that are not error level log events for the overall program. You could say that an operation error is anything that prevents an operation from completing successfully, while a pro…

This is why it’s almost always wrong for library functions to log anything, even on ”errors”. Pass the status up through return values or exceptions. As a library author you have no clue as how an application might use it. Multi threading, retry loops and expected failures will turn what’s a significant event in one context into what’s not even worthy of a debug log in another. No rule without exceptions of course, o…

On paper, USDT probes are the best way for libraries (and binaries) to provide information for debugging because they can be used programmatically and have no performance overhead until they are measured but unfortunately they are not widely used.

Re: Log level 'error' should mean that something needs to be fixed

#202

Post by tsimionescu »

01HNNWZ0MV43FF wrote: post #155
TZubiri wrote: post #139

Earlier quoted context omitted.

How about wrapping the log.trace param in a lambda and monkeypatching log.trace to take a function that returns a string, and of course pushing the conditional to the monkeypatched func.

That is why the popular `tracing` crate in Rust uses macros for logging instead of functions. If the log level is too low, it doesn't evaluate the body of the macro

Does that mean the log level is a compilation parameter? Ideally, log levels shouldn't even be startup parameters, they should be changeable on the fly, at least for any server side code. Having to restart if bad enough, having to recompile to get debug logs would be an extraordinary nightmare (not only do you need to get your customers to reproduce the issue with debug logs, you actually have to ship them new binaries, which likely implies export controls and security validations etc).

Re: Log level 'error' should mean that something needs to be fixed

#203

Post by vips7L »

esrauch wrote: post #185
ivan_gammel wrote: post #157

Earlier quoted context omitted.

> library has existed for a decade >but Java removed a method that let you make it fast, but you can still run slow without that API I’d like to see an example of that, because this is extremely hypothetical scenario. I don’t think any library is so advanced to anticipate such scenarios and write something to log. And of course Java specifically has longer cycle of deprecation and removal. :) As for your second examp…

Protobuf is the example I had in mind. It uses sun.misc.Unsafe which is being removed in upcoming Java releases, but it has a slow fallback path. It logs a warning when it runs if it can tell it's only using the fallback path but the fast path is still available if the application owner set a flag to turn it back on if they want to: https://github.com/protocolbuffers/protobuf/issues/20760 Java Protobuf also logs a wa…

What’s stopping you from using the replacements provided in VarHandle and MemorySegment? Just wanting to support the 10 year old JDK 8?

Re: Log level 'error' should mean that something needs to be fixed

#204

Post by [deleted] »

Xss3 wrote: post #183

Some programs are error resistant and need an additional level: Fatal. A warning can be ignored safely. Warnings may be 'debugging enabled, results cannot be certified' or something similar. An error should not be ignored, an operation is failing, data loss may be occurring, etc. Some users may be okay with that data loss or failing operation. Maybe it isnt important to them. If the program continues and does not err…

[deleted]

Re: Log level 'error' should mean that something needs to be fixed

#205

Post by bluGill »

tsimionescu wrote: post #202
01HNNWZ0MV43FF wrote: post #155

Earlier quoted context omitted.

That is why the popular `tracing` crate in Rust uses macros for logging instead of functions. If the log level is too low, it doesn't evaluate the body of the macro

Does that mean the log level is a compilation parameter? Ideally, log levels shouldn't even be startup parameters, they should be changeable on the fly, at least for any server side code. Having to restart if bad enough, having to recompile to get debug logs would be an extraordinary nightmare (not only do you need to get your customers to reproduce the issue with debug logs, you actually have to ship them new binari…

I don't know how rust does it, but my internal C++ framework has a global static array so that we can lookup the current log level quickly, and change it at runtime as needed. It is very valuable to turn on specific debug logs at times, when someone has a problem and we want to know what some code is doing

Re: Log level 'error' should mean that something needs to be fixed

#206

Post by esrauch »

vips7L wrote: post #203
esrauch wrote: post #185

Earlier quoted context omitted.

Protobuf is the example I had in mind. It uses sun.misc.Unsafe which is being removed in upcoming Java releases, but it has a slow fallback path. It logs a warning when it runs if it can tell it's only using the fallback path but the fast path is still available if the application owner set a flag to turn it back on if they want to: https://github.com/protocolbuffers/protobuf/issues/20760 Java Protobuf also logs a wa…

What’s stopping you from using the replacements provided in VarHandle and MemorySegment? Just wanting to support the 10 year old JDK 8?

There's a lot here, to be honest these things always come back to investment cost and ROI compared to everything else that could be worked on.

Java 8 is still really popular, probably the most popular single version. It's not just servers in context, but also Android where Java 8 is the highest safe target, it's not clear what decade we'll be in when VarHandle would be safe to use there at all.

VarHandle was Java 9 but MemorySegment was Java 17. And the rest of FFM is only in 25 which is fully bleeding edge.

Protobuf may realistically try to move off of sun.misc.unsafe without the performance regressions in a way that is without adopting MemorySegment to avoid the versioning problem, but it takes significant and careful engineering time.

That said it's always possible to have waterfall of preferred implementations based on what's supported, it's just always an implementation/verification costs.

Re: Log level 'error' should mean that something needs to be fixed

#207

Post by lanstin »

Copenjin wrote: post #164
rsanek wrote: post #160

If something needs to be fixed, why is it just a log? How is someone supposed to even notice a random error log? At the places that I've worked, trying to make alerting be triggered on only logs was always quite brittle, it's just not best practice. Throw an exception / exit the program if it's something that actually needs fixing!

> If something needs to be fixed, why is it just a log? What he meant is that is an unexpected condition, that should have never happened, but that did, so it needs to be fixed. > How is someone supposed to even notice a random error log? Logs should be monitored. > At the places that I've worked, trying to make alerting be triggered on only logs was always quite brittle, it's just not best practice. Because the logs…

While it is fun to have your code run for 500 days without restart, it is a bad architecture. You should be able to move load around from host to host or network to network without losing any work. This involves graceful draining and then shutting down the old.

For impossible errors exiting and sending the dev team as much info as possible (thread dump, memory dump, etc) is helpful.

In my experience logs are good for finding out what is wrong once you know something is wrong. Also if the server is written to have enough but not too much logging you can read them over and get a feel for normal operation.

Re: Log level 'error' should mean that something needs to be fixed

#208

Post by lanstin »

electroly wrote: post #37
raldi wrote: post #10

Earlier quoted context omitted.

No; I’m not understanding your point about guessing. Could you restate? As for queries that time out, that should definitely be a metric, but not pollute the error loglevel, especially if it’s something that happens at some noisy rate all the time.

I think OP is making two separate but related points, a general point and a specific point. Both involve guessing something that the error-handling code, on the spot, might not know. 1. When I personally see database timeouts at work it's rarely the database's fault, 99 times out of 100 it's the caller's fault for their crappy query; they should have looked at the query plan before deploying it. How is the error-hand…

Also everywhere I have worked there are transient network glitches from time to time. Timeout can often be caused by these.

Re: Log level 'error' should mean that something needs to be fixed

#209

Post by lanstin »

danaris wrote: post #109
ImPostingOnHN wrote: post #32

Earlier quoted context omitted.

That is a lagging indicator. By the time you're alerted, you've already failed by letting users experience an issue.

Well, yes . If the cable falls out of the server (or there's a power outage, or a major DDoS attack, or whatever), your users are going to experience that before you are aware of it. Especially if it's in the middle of the night and you don't have an active night shift. Expecting arbitrary services to be able to deal with absolutely any kind of failure in such a way that users never notice is deeply unrealistic.

It continues to become more realistic with the passing of time.

Re: Log level 'error' should mean that something needs to be fixed

#210

Post by lanstin »

raldi wrote: post #5
shadowgovt wrote: post #2

This is the standard I use as well. In general, my rule of thumb is that if something is logging error, it would have been perfectly reasonable for the program to respond by crashing, and the only reason it didn't is that it's executing in some kind of larger context that wants to stay up in the event of the failure of an individual component (like one handler suffering a query that hangs it and having to be terminat…

Right. If staging or the canary is logging errors, you block/abort the deploy. If it’s logging warnings, that’s normal.

Unless it is logging more warnings because your new code is failing somehow; maybe it stopped parsing the reply correctly from a "is this request rate limited" service so it is only returning 429 to callers never accepting work.
Post reply on HN