Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Log level 'error' should mean that something needs to be fixed

utcc.utoronto.ca

Post reply on HN
Print view View original

301–310 of 313 posts

Re: Log level 'error' should mean that something needs to be fixed

#301

Post by tsimionescu »

TZubiri wrote: post #291
ekidd wrote: post #273

Earlier quoted context omitted.

In my professional life, somewhere over 99% of time, the code suffering the error has either been: 1. Production code running somewhere on a cluster. 2. Released code running somewhere on a end-user's machine. 3. Released production code running somewhere on an end-user's cluster. And errors happen at weird times, like 3am on a Sunday morning on someone else's cluster. So I'd just as soon not have to wake up, figurin…

I think I get the idea, gdb is too powerful. For contexts where operator is distinct from manufacturer, the debug/logging tool needs to be weaker and not ad-hoc so it can be audited and to avoid exfiltrating user data.

It's not so much about power, but about the ad-hoc nature of attaching a debugger. If you're not there to catch and treat the error as it happens, a debugger is not useful in the slightest: by the time you can attach it, the error, or the context where it happened, are long gone. Not to mention, even if you can attach a debugger, it's most often not acceptable to pause the execution of the entire process for you to debug the error.

Especially since a lot of the time an exception being raised is not the actual bug: the bug happened many functions before. By logging key aspects of the state of the program, even in non-error cases, when an error happens, you have a much better chance of piecing together how you got to the error state in the first place.

Re: Log level 'error' should mean that something needs to be fixed

#302

Post by TZubiri »

dpark wrote: post #240
TZubiri wrote: post #230

Earlier quoted context omitted.

I was just trying to think of an example of a non idempotent function. As in it depends on an external IO device. I will say that error handling and logging in general is one of my weakpoints, but I made a comment about my approach so far being dbg/pdb based, attaching a debugger and creating breakpoints and prints ad-hoc rather than writing them in code. I'm sure there's reasons why it isn't used as much and logging…

The issue with relying on gdb is that you generally cannot do this in production. You can’t practically attach a debugger to a production instance of a service for both performance and privacy reasons, and the same generally applies to desktop and mobile applications being run by your customers. Gdb is mostly for local debugging and the truth is that “printf debugging” is how it often works for production. (Plus exce…

It's coming together more clearly now.

I guess my experience is more from the role of a startup or even in-house software. So we both develop and operate the software. But in scenarios where you ship the software and it's operated by someone else, it makes sense to have more auditable and restricted logging instead of all-too-powerful ad-hoc debugging.

Re: Log level 'error' should mean that something needs to be fixed

#303

Post by jedberg »

Joker_vD wrote: post #250
jedberg wrote: post #154

I feel like it's more nuanced than OP writes. Presumably every log line comes from something like a try/catch. An edge case was identified, and the code did something differently. Did it do what it was supposed to do, but in a different way or defer for retrying later? Then WARN. Did it fail to do what it needed to do? ERROR Did it do what it needed to do in the normal way because it was totally recoverable? INFO Did…

> Did it do what it was supposed to do, but in a different way or defer for retrying later? Then WARN. > Did it fail to do what it needed to do? ERROR > Did it do what it needed to do in the normal way because it was totally recoverable? INFO We have a web-facing system (it uses a custom request-response protocol on top of Websocket... it's an old system) that users are routinely trying to, ahem, automate even though…

It sounds like it did exactly what it was supposed to do -- reject the bad input. Looks like an INFO to me.

Re: Log level 'error' should mean that something needs to be fixed

#304

Post by lelanthran »

awesome_dude wrote: post #181
Too wrote: post #153

Earlier quoted context omitted.

Simple: include those relevant details in the exceptions instead of hiding them.

At the extreme end: If my Javascript frontend is being told about a database configuration error happening in the backend when a call with specific parameters is made - that is a SERIOUS security problem. Errors are massaged for the reader - a database access library will know that a DNS error occurred and that is (the first step for debugging) why it cannot connect to the specified datastore. The service layer calle…

> At the extreme end: If my Javascript frontend is being told about a database configuration error happening in the backend when a call with specific parameters is made - that is a SERIOUS security problem.

I'll accept that it is a security problem; why would it be a serious security problem? Any error that the client knows about the configuration is unlikely to be one that is exploitable anyway, and if it is (for example, the client gets told "could not connect to 192.168.1.139:5432"), then you have bigger problems than sending error messages to clients.

What sort of example did you have in mind that makes this a serious security problem?

Re: Log level 'error' should mean that something needs to be fixed

#305

Post by cpburns2009 »

srdjanr wrote: post #237
cpburns2009 wrote: post #72

Earlier quoted context omitted.

It really depends on the third party service. For service A, a 500 error may be common and you just need to try again, and a descriptive 400 error indicates the original request was actually handled. In these cases I'd log as a warning. For service B, a 500 error may indicate the whole API is down, in which case I'd log a warning and not try any more requests for 5 minutes. For service C, a 500 error may be an anomal…

What's the difference between B and C? API being down seems like an anomaly. Also, you can't know how frequently you'll get 500s at the time you're doing integration, so you'll have to go back after some time to revisit log severities. Which doesn't sound optimal.

I forgot to mention that for service B, the API being down is a common, daily occurrence and does not last long. The behavior of services A-C is from my real world experience.

I do mean revisiting the log seventies as the behavior of the API becomes known. You start off treating every error as a hard error. As you learn the behavior of the API over time, you adjust the logging and error handling accordingly.

Re: Log level 'error' should mean that something needs to be fixed

#306

Post by just6979 »

buggjenrmf wrote: post #251
rr808 wrote: post #225

Earlier quoted context omitted.

Still quite like the windows log approach which (if logged) stores the template as just the id, with the values, saving lots of storage as well eg 123, foo, bar. You can concatenate in the reader.

So, it costs perf every time it’s read, instead of when it’s written (once). And of course has a lot of overhead to store metadata. Bad design. As usual.

Except it's always written, but almost never read. Something that is fast/non-resource-intensive to write is definitionally a better design for logging.

What metadata? The raw template? That's data in this case, data for the later rendering of logs. Yes, the template plus the params is going to be slightly bigger than a rendered string, but that's the speed/size tradeoff inherent almost everywhere. It may even keep seperate things like the subsystem, event type, log level, etc; which trades off size (again) for speed/ease of filtering. It's all trade-offs, and to blanket declare one method (the Windows method in this case) as just bad design is only displaying your own ignorance, or bias.

Re: Log level 'error' should mean that something needs to be fixed

#307

Post by oofbey »

georgefrowny wrote: post #264
oofbey wrote: post #89

Earlier quoted context omitted.

At scale the rare events start to happen reliably. Hardware failures almost certainly cause ERROR conditions. Network glitches. Our production system pages oncall for any errors. At night it will only wake somebody up for a whole bunch of errors. This discipline forces us to take a look at every ERROR and decide if it is spurious and out of our control or something we can deal with. At some point our production syste…

I think if someone is going be gotten out of bed that would be a critical rather then error. Generally I'd say in a large "live" system, errors end up raising Jira tickets, criticals end up ringing phones.

Most systems I’ve worked with can go completely offline without ever logging a critical error. Some coding errors or misconfiguration or failure in a critical system - enough to log an error - and nobody can get any useful work done. I’ve never seen sobering that cash convert those into critical errors. I’m used to critical errors being rare - certain failures of a server to start. Or infra problems.

Re: Log level 'error' should mean that something needs to be fixed

#308

Post by awesome_dude »

lelanthran wrote: post #304
awesome_dude wrote: post #181

Earlier quoted context omitted.

At the extreme end: If my Javascript frontend is being told about a database configuration error happening in the backend when a call with specific parameters is made - that is a SERIOUS security problem. Errors are massaged for the reader - a database access library will know that a DNS error occurred and that is (the first step for debugging) why it cannot connect to the specified datastore. The service layer calle…

> At the extreme end: If my Javascript frontend is being told about a database configuration error happening in the backend when a call with specific parameters is made - that is a SERIOUS security problem. I'll accept that it is a security problem; why would it be a serious security problem? Any error that the client knows about the configuration is unlikely to be one that is exploitable anyway , and if it is (for e…

2. Verbose Error Messages: When Your Application Talks Too Much Verbose error messages represent another common misconfiguration that gifts critical information to attackers. When applications encounter errors, they often generate detailed messages intended for developers. In production environments, these messages can reveal:

Technical infrastructure details: Database types, versions, server configurations File paths and directory structures: Enabling directory traversal attacks Programming logic: Including code snippets that expose application behavior Sensitive credentials: Database connection strings, usernames, passwords Software versions: Allowing attackers to identify known vulnerabilities The impact of this vulnerability is significant. Error messages can expose not just that a system runs PHP, but that it runs a specific, unsupported version — providing attackers with a clear exploitation path.

Security researchers have documented numerous instances where verbose error messages enabled breaches:

Dating App Vulnerability (2016): Tinder’s login system displayed error messages indicating whether specific email addresses were registered, enabling brute-force attacks to identify valid accounts. Password Manager Leak (2019): A popular password manager’s login form disclosed through error messages whether email addresses were registered with the service, facilitating targeted attacks. Government Agency Breach (2020): A major US government agency’s website displayed error messages revealing whether specific usernames existed in the system, enabling attackers to enumerate valid accounts.

[1] https://medium.com/@instatunnel/security-misconfiguration-th...

Re: Log level 'error' should mean that something needs to be fixed

#309

Post by lelanthran »

awesome_dude wrote: post #308
lelanthran wrote: post #304

Earlier quoted context omitted.

> At the extreme end: If my Javascript frontend is being told about a database configuration error happening in the backend when a call with specific parameters is made - that is a SERIOUS security problem. I'll accept that it is a security problem; why would it be a serious security problem? Any error that the client knows about the configuration is unlikely to be one that is exploitable anyway , and if it is (for e…

2. Verbose Error Messages: When Your Application Talks Too Much Verbose error messages represent another common misconfiguration that gifts critical information to attackers. When applications encounter errors, they often generate detailed messages intended for developers. In production environments, these messages can reveal: Technical infrastructure details: Database types, versions, server configurations File path…

First, I disagree that "user emails can be brute-forced" is a serious security issue.

I mean, sure, it's a security issue, but on a scale of 1-10, with 1 being "security issue, we'll fix in next point release" and 10 being "All-hands until this emergency patch goes out, and we keep the system offline while fixing it", this is definitely a 1.

Secondly, this barely counts as a security issue; some systems I worked on recently required error messages to tell the user how to fix the error they got. You don't simply say (for example) "attachment not found", you say "Field $FIELD is empty. This is a mandatory field" or similar.

There are still plenty of secure systems out there that will direct the user to create an account if an unregistered user attempts to log in.

It's a trade-off in usability: some places go the "Authentication failed (but we won't tell you why)" route, and others go the "Click here to sign up" route.

Re: Log level 'error' should mean that something needs to be fixed

#310

Post by lanstin »

dolmen wrote: post #228
lanstin wrote: post #211

Earlier quoted context omitted.

If they cause your customers to ditch your product but calling them and saying "your calls are all getting 4xx because you are not putting the state code into the call parameters" would keep them as customers, then you would be wise to make that communication.

But first ensure that the input error is properly reported to the client in the response body (ideally in a structured way), so the client could have figured out by himself. If a fix is needed on your side for this matter, having a conversation with a customer might be useful before breaking more stuff. ("We have no state code in EU. Why is that mandatory?").

If you are trying to sell a product, it is sometimes useful to solve people problems for them, rather than counting on them to figure them out on their own.
Post reply on HN