Why debugging is all about understanding

Sometimes that is true; if your architecture is suboptimal then yes you can trim away huge classes of bugs with a high-level change. But as with all aspects of software development, the truth is a question of nuance and judgement.

The way your assertion can go wrong is when you don't recognize the errors that your rewritten code will add. At a high level we always envisage perfect code, and existing code is always seemingly full of warts, but no perfect coding plan ever survives contact with reality. Those warts are bug fixes and optimizations which make the thing work. Don't be too quick to rewrite lest you create bigger problems than you started with.

Wait a minute. First of all, the data is simply "there is a reproducible bug in the program". The real hypothesis is that a small, localized portion of the code is responsible for the defect. To find it, you can try binary search. If you have a better idea of the location, then by all means.

Second, when you have a bug and you write a unit test, you are effectively commenting out the entire codebase except for the function under test. When you have a compiler error, whether it's a syntax/semantics bug in your code or a bug in the compiler, sometimes you need to produce a minimal example, so you have to cut cut cut until the bug is just barely provoked. When you have a pipeline of data transforms, and the end result is suddenly borked, it can work to chop off half the transforms and look at the result. When latex is crashing for some unintelligible reason, just comment out half of your document and see if the problem goes away.

Sure it's really dumb if you're just excluding a* .c through m* .c (spaces due to HN formatting rules), but figuring out if the problem is in the first or second half of main is not outrageous. I don't think the guy was presenting it as the first step ("If you have no idea where your bug lives"), but I do agree that it comes across as a little naive, since he should have talked about all of the other techniques available first. So the problem isn't so much the lack of a hypothesis, but the inefficient experimental approach of using a brute force technique indiscriminately.

I think the last resort is reached a little sooner for some types of bugs and some experience levels (language, environment, codebase, programming), and yes in many cases it won't even do anything for you.

Personally I always liked dtrace. This guy gave a demo of it at my university once, I thought it was great, one of the best talks I've seen.

Regarding 7, I think you should explicitly mention "git bisect" and similar tools. (Or, are you doing that step by hand?!)

For me, when I run out of ideas, "git bisect" is the most efficient debugging tool. On success, it allows me to pin down the exact commit that introduced the bug. It is very fast to do (binary search), and on success it narrows the bug down to a relatively small amount code to check.

Of course, this only works if you have a clean code history: small commits and always a working state after each commit. That is, no intermediate crap commits, and no monster commits doing 3 things at once.

IMHO, precious moment when bug is found in the code AND it is not obvious, so you need to run debugger, is perfect time to improve your code. It indicates that code is complicated, or not well covered, or poorly commented (because bug is not noticed at peer review). You are going to test this piece of code anyway, because you need to check is bug fixed, so why not to spare some time to make code better and battle-test your improvement: if you catch bug in the process, then you improved code, if not: keep going.

And no, I am not talking about asserts. Assertions are only small part of real debugger.

Typical tools I use in my own programs are: good logging and tracing, which are able to explain what happens inside program without spamming me, some test cases for critical parts and for fixed bugs. Often, I will also split large program or script into smaller independent parts, with their own options, configuration, loggind and test cases. It's all.

I don't mean to speak for bcantrill, but yes, this is exactly how we (at Joyent) build programs. We wrote in detail about why we do it this way: https://www.joyent.com/developers/node/design/errors

It's not that this approach motivates root-causing failures (though that's true). It's that uncaught exceptions are programmer errors, and it's tautologically impossible to correctly deal with them. Attempting to do so can make things much, much worse.

To make this concrete: I've dealt more than one production outage caused by the mere presence of an uncaughtException handler. If the program had merely crashed, a hundred requests may have been interrupted, but the program would have restarted and resumed servicing requests. Instead, the exception was thrown from a code path that had a database transaction open with database locks held. Because the uncaughtException handler just logged the exception and otherwise ignored it, that transaction stayed open (and the locks remained held) until a human got into the loop -- interfering with tens of thousands of subsequent requests. That's much, much worse. If the process had just exited, the db connection would have been closed, the transaction aborted, and the locks released.

An unexpected error handler can't know the complete state that the program was in because by definition this wasn't a state that the programmer thought about.