Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging: Indispensable rules for finding even the most elusive problems (2004)

dwheeler.com

Post reply on HN
Print view View original

161–170 of 238 posts

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#161

Post by gregthelaw »

tetha wrote: post #118
throwawayfks wrote: post #110

Earlier quoted context omitted.

I read this book and took this advice to heart. I don't have a brass bar in the office, but when I'm about to push a button that could cause destructive changes, especially in prod, my hands reflexively fly up into the air while I double-check everything.

A weird, yet effective recommendation from someone at my last job: If it's a destructive or dangerous action in prod, touch both your elbows first. This forces ou to take the hands away from the keyboard, stop any possible auto-pilot and look what you're doing.

Related: write down what you're seeing (or rather, what you _think_ you're seeing), and so with pen and paper, not the keyboard. You can type way faster than you can write, and the slowness of writing makes you think harder about what you think you know. Often you do the know the answer, you just have to tease it out. Or there are gaps in your knowledge that you hadn't clocked. After all, an assumption is something you don't realise you've made.

This also works well in conjunction with debug tooling -- the tooling gives you the raw information, writing down that information helps join the dots.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#162

Post by pcblues »

Over twenty five odd years, I have found the path to a general debugging prowess can best be achieved by doing it. I'd recommend taking the list/buying the book, using https://up-for-grabs.net to find bugs on github/bugzilla, etc. and doing the following:

1. set up the dev environment

2. fork/clone the code

3. create a new branch to make changes and tests

4. use the list to try to find the root cause

5. create a pull request if you think you have fixed the bug

And use Rule 0 from GuB-42: Don't panic

(edited for line breaks)

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#163

Post by samsquire »

One thing I have been doing is to create a directory called "debug" from the software and write lots of different files when the main program has executed to add debugging information but only write files outside of hot loops for debugging and then visually inspect the logs when the program is exited.

For intermediate representations this is better than printf to stdout

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#164

Post by seanwilson »

n144q wrote: post #144
seanwilson wrote: post #43

Earlier quoted context omitted.

I don't think this is always worth it. Some tests can be time consuming or complex to write, have to be maintained, and we accept that a test suite won't be testing all edge cases anyway. A bug that made it to production can mean that particular bug might happen again, but it could be a silly mistake and no more likely to happen again than 100s of other potential silly mistakes. It depends, and writing tests isn't fr…

You (or other people) will thank yourself in a few months/years when refactoring the code, knowing that they don't need to worry about missing edge cases, because all known edge cases are covered with these non regression tests.

There's really no situation you wouldn't write a test? Have you not had situations where writing the test would take a lot of effort vs the risk/impact of the bug it's checking for? Your test suite isn't going to be exhaustive anyway so there's always a balance of weighing up what's worth testing. Going overkill with tests can actually get in the way of refactoring as well when a change to the UI or API isn't done because it would require updating too many tests.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#165

Post by adolph »

nox101 wrote: post #153

> #1 Understand the system: Read the manual, read everything in depth, know the fundamentals, know the road map, understand your tools, and look up the details. Maybe I'm mis-understand but "Read the manual, read everything in depth" sounds like. Oh, I have bug in my code, first read the entire manual of the library I'm using, all 700 pages, then read 7 books on the library details, now that a month or two has passed…

This was written in 2004, the year of Google's IPO. Atwood and Spolsky didn't found Stack Overflow until 2008. [0] People knew things as the "Camel book" [1] and generally just knew things.

0. https://stackoverflow.blog/2021/12/14/podcast-400-an-oral-hi...

1. https://www.perl.com/article/extracting-the-list-of-o-reilly...

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#166

Post by gregthelaw »

ajross wrote: post #83
nickjj wrote: post #33

For #4 (divide and conquer), I've found `git bisect` helps a lot. If you have a known good commit and one of dozens or hundreds of commits after that is bad, this can help you identify the bad commit / code in a few steps. Here's a walk through on using it: https://nickjanetakis.com/blog/using-git-bisect-to-help-find... I jumped into a pretty big unknown code base in a live consulting call and we found the problem pr…

Not to complain about bisect, which is great. But IMHO it's really important to distinguish the philosophy and mindspace aspect to this book (the "rules") from the practical advice ("tools"). Someone who thinks about a problem via "which tool do I want" (c.f. "git bisect helps a lot"[1]) is going to be at a huge disadvantage to someone else coming at the same decisions via "didn't this used to work?"[2] The world is…

I've spent the past two decades working on a time travel debugger so obviously I'm massively biassed, but IMO most programmers are not nearly as proficient in the available debug tooling as they should be. Consider how long it takes to pick up a tool so that you at least have a vague understanding of what it can do, and compare to how much time a programmer spends debugging. Too many just spend hour after hour hammering out printf's.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#167

Post by gregthelaw »

forrestthewoods wrote: post #135
michalsustr wrote: post #116

Earlier quoted context omitted.

This is why we use squash like here https://docs.gitlab.com/ee/user/project/merge_requests/squas...

I explicitly don’t want squash. The commits are still worth keeping separate. There’s lots of distinct pieces of work. But sometimes you break something and fix it later. Or you add something new but support different environments/platforms later.

But if you don't squash, doesn't this render git bisect almost useless?

I think every commit that gets merged to main should be an atomic believed-to-work thing. Not only does this make bisect way more effective, but it's a much better narrative for others to read. You should write code to be as readable by others as possible, and your git history likewise.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#168

Post by gregthelaw »

soco wrote: post #21
david_draco wrote: post #13

Step 10, add the bug as a test to the CI to prevent regressions? Make sure the CI fails before the fix and works after the fix.

What do you do with the years old bug fixes? How fast can one run the CI after a long while of accumulating tests? Do they still make sense to be kept in the long run?

I would say yes, your CI should accumulate all of those regression tests. Where I work we now have many, many thousands of regression test cases. There's a subset to be run prior to merge which runs in reasonable time, but the full CI just cycles through.

For this to work all the regression tests must be fast, and 100% reliable. It's worth it though. If the mistake was made once, unless there's a regression test to catch it, it'll be made again at some point.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#169

Post by sroussey »

o_nate wrote: post #150
GuB-42 wrote: post #48

Rule 0: Don't panic Really, that's important. You need to think clearly, deadlines and angry customers are a distraction. That's also when having a good manager who can trust you is important, his job is to shield you from all that so that you can devote all of your attention to solving the problem.

A corollary to this is always have a good roll-back plan. It's much nicer to be able to roll-back to a working version and then be able to debug without the crisis-level pressure.

Rollback ability is a must—it can be the most used mitigation if done right.

Not all issues can be fixed with a rollback though.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#170

Post by Noumenon72 »

aag wrote: post #137
jerf wrote: post #57

Earlier quoted context omitted.

"git bisect" is why I maintain the discipline that all commits to the "real" branch, however you define that term, should all individually build and pass all (known-at-the-time) tests and generally be deployable in the sense that they would "work" to the best of your knowledge, even if you do not actually want to deploy that literal release. I use this as my #1 principle, above "I should be able to see every keystrok…

Sometimes you'll find a repo where that isn't true. Fortunately, git bisect has a way to deal with failed builds, etc: three-value logic. The test program that git bisect runs can return an exit value that means that the failure didn't happen, a different value that means that it did, or a third that means that it neither failed nor succeeded. I wrote up an example here: https://speechcode.com/blog/git-bisect

Very handy! I forgot about `git bisect skip`.
Post reply on HN