Live data from Hacker News

You say “cave dweller debugging”, I say debug logging

sicpers.info

51–60 of 139 posts

Re: You say “cave dweller debugging”, I say debug logging

#51
post #22

You can't debug a program that ran in the past, but you can read its info-level logs.

Depends on the program.

In critical systems, it's not unheard of to transaction log every input and every response to external requests, have strictly deterministic code, so that you specifically can replay or reconstruct a system state.

It's a pretty demanding way to build software, but you also get extremely resilient software as a result.

Re: You say “cave dweller debugging”, I say debug logging

#52
A lot of logging practices do seem to orbit the idea that we should be able to handle this in a more automated fashion - i.e. we're missing a suitable zero-cost/minimal cost abstraction.

In an ideal world, without restart, an application would let me selectively switch on logging levels from "errors only" up to "line-level original source code in a targeted function" - with a trigger system.

Functionally basically a debugger, but coded into the binary so we don't pay the debugger cost (and also so we can try it more broadly across a cluster). This is the direction that Linux kernel eBPF debugging is really leading us, and what it should be easier to get that sort of power into our applications at a code-size but not speed cost - even if it means having our compilers just include multiple copies of the function with different logging-level functionality enabled.

Re: You say “cave dweller debugging”, I say debug logging

#53

> Instead, call your syslog/OSLog/logger function, with an appropriate severity level (probably DEBUG) and some easily filterable preamble Good patterns for JS here? Maybe console.debug() or console.warn()?

Not sure what the best practice is, but probably you don't want console.debug -- IIRC debug is logged by default, so unless you strip console logs in prod (which defeats the purpose), your users will have their console spammed.

Re: You say “cave dweller debugging”, I say debug logging

#55
post #22

You can't debug a program that ran in the past, but you can read its info-level logs.

You can debug programs that ran in the past using debuggers like rr[0], which support both recording execution for later debugging, or stepping backward in a running process.

[0]: https://rr-project.org/

Re: You say “cave dweller debugging”, I say debug logging

#56
Yes, debug logging is a great way to encode educated guesses about where unexpected behavior will appear in the future, in a way that should save time on net when a guess is correct.

But one should be aware of its limits. Debugging often involves performing something like a binary search over a collection of suspicious assumptions. The first few well-placed debug-logging messages can save several iterations in these binary searches. But the cost of saving one more binary search iteration in this manner increases exponentially. As with so many other things in engineering, moderation is a virtue; you want good debug-logging and an ability to perform quick manual tweaks to the code to cover the "last mile" of a debugging session, when this is at all possible in your execution environment.

Re: You say “cave dweller debugging”, I say debug logging

#58
post #29

Earlier quoted context omitted.

> Making logs during development also forces developers to make a conscious effort of finding a balance of what and when to log something, create proper debug levels not to overwhelm the logs, while still providing some useful information... The need for this balance is why I don't completely agree with the article. If I'm debugging something with log messages, I put way more detail into it than I want to see in prod…

I don't know that there's an easy solution to this problem, but what I've seen Kubernetes and other Google-derived projects do is replace standard log levels with an increasing series of integers so you can log in increasing detail. This seems a little arbitrary (one developer's 8 might be another developer's 14 or something) but at least you can easily ask for increasingly more detail until you get what you need wit…

The increased detail also clutters the source, though.

As I write this I realise that the "right" solution is that my IDE also folds away log calls of below some settable priority... Then I need a linter to prevent any code with side effects from running in a log line... Then I need to work in a language where that kind of static analysis is definitely possible... Maybe it's not so simple.

Re: You say “cave dweller debugging”, I say debug logging

#59
post #30

For my own code, 90% of the bugs are caught using logging and assert. The remaining are caught via debuggers. For working on unfamiliar code, using a debugger to step through the logic helps to understand the flow to find the problem.

And the remaining remaining bugs are not caught.

Re: You say “cave dweller debugging”, I say debug logging

#60
post #29

Earlier quoted context omitted.

> Making logs during development also forces developers to make a conscious effort of finding a balance of what and when to log something, create proper debug levels not to overwhelm the logs, while still providing some useful information... The need for this balance is why I don't completely agree with the article. If I'm debugging something with log messages, I put way more detail into it than I want to see in prod…

I don't know that there's an easy solution to this problem, but what I've seen Kubernetes and other Google-derived projects do is replace standard log levels with an increasing series of integers so you can log in increasing detail. This seems a little arbitrary (one developer's 8 might be another developer's 14 or something) but at least you can easily ask for increasingly more detail until you get what you need wit…

This just creates a festering problem because devs are given too much freedom. A better solution is just two levels, basic and verbose, with subsystem feature flags you can control to prevent verbose from being a fire hose.
Post reply on HN