Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging: Indispensable rules for finding even the most elusive problems (2004)

dwheeler.com

Post reply on HN
Print view View original

141–150 of 238 posts

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#141

Post by hughdbrown »

In my experience, the most pernicious temptation is to take the buggy, non-working code you have now and to try to modify it with "fixes" until the code works. In my experience, you often cannot get broken code to become working code because there are too many possible changes to make. In my view, it is much easier to break working code than it is to fix broken code.

Suppose you have a complete chain of N Christmas lights and they do not work when turned on. The temptation is to go through all the lights and to substitute in a single working light until you identify the non-working light.

But suppose there are multiple non-working lights? You'll never find the error with this approach. Instead, you need to start with the minimal working approach -- possibly just a single light (if your Christmas lights work that way), adding more lights until you hit an error. In fact, the best case is if you have a broken string of lights and a similar but working string of lights! Then you can easily swap a test bulb out of the broken string and into the working chain until you find all the bad bulbs in the broken string.

Starting with a minimal working example is the best way to fix a bug I have found. And you will find you resist this because you believe that you are close and it is too time-consuming to start from scratch. In practice, it tends to be a real time-saver, not the opposite.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#142

Post by CobrastanJorji »

GuB-42 wrote: post #48

Rule 0: Don't panic Really, that's important. You need to think clearly, deadlines and angry customers are a distraction. That's also when having a good manager who can trust you is important, his job is to shield you from all that so that you can devote all of your attention to solving the problem.

I once worked for a team that, when a serious visible incident occurred, a company VP would pace the floor, occasionally yelling, describing how much money we were losing per second (or how much customer trust if that number was too low) or otherwise communicating that we were in a battlefield situation and things were Very Critical.

Later I worked for a company with a much bigger and more critical website, and the difference in tone during urgent incidents was amazing. The management made itself available for escalations and took a role in externally communicating what was going on, but besides that they just trusted us to do our jobs. We could even go get a glass of water during the incident without a VP yelling at us. I hadn't realized until that point that being calm adults was an option.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#144

Post by n144q »

seanwilson wrote: post #43
david_draco wrote: post #13

Step 10, add the bug as a test to the CI to prevent regressions? Make sure the CI fails before the fix and works after the fix.

I don't think this is always worth it. Some tests can be time consuming or complex to write, have to be maintained, and we accept that a test suite won't be testing all edge cases anyway. A bug that made it to production can mean that particular bug might happen again, but it could be a silly mistake and no more likely to happen again than 100s of other potential silly mistakes. It depends, and writing tests isn't fr…

You (or other people) will thank yourself in a few months/years when refactoring the code, knowing that they don't need to worry about missing edge cases, because all known edge cases are covered with these non regression tests.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#145

Post by snowfarthing »

michalsustr wrote: post #116
forrestthewoods wrote: post #93

Earlier quoted context omitted.

> why I maintain the discipline that all commits to the "real" branch, however you define that term, should all individually build and pass all (known-at-the-time) tests and generally be deployable in the sense that they would "work" to the best of your knowledge, even if you do not actually want to deploy that literal release You’re spot on. However it’s clearly a missing feature that Git/Mercurial can’t tag diffs a…

This is why we use squash like here https://docs.gitlab.com/ee/user/project/merge_requests/squas...

As someone who doesn't like to see history lost via "rebase" and "squashing" branches, I have had to think through some of these things, since my personal preferences are often trampled on by company policy.

I have only been in one place where "rebase" is used regularly, and now that I'm a little more familiar with it, I don't mind using it to bring in changes from a parent branch into a working branch, if the working branch hasn't been pushed to origin. It still weirds me out somewhat, and I don't see why a simple merge can't just be the preferred way.-

I have, however, seen "squashing" regularly (and my current position uses it as well as rebasing) -- and I don't particularly like it, because sometimes I put in notes and trials that get "lost" as the task progresses, but nonetheless might be helpful for future work. While it's often standard to delete "squashed" branches, I cannot help but think that, for history-minded folks like me, a good compromise would be to "squash and keep" -- so that the individual commits don't pollute the parent branch, while the details are kept around for anyone needing to review them.

Having said that, I've never been in a position where I felt like I need to "forcibly" push for my preferences. I just figure I might as well just "go with the flow", even if a tiny bit of me dies every time I squash or rebase something, or delete a branch upon merging!

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#146

Post by n144q »

jerf wrote: post #61
Ragnarork wrote: post #49

Earlier quoted context omitted.

Writing tests isn't free but writing non-regression tests for bugs that were actually fixed is one of the best test cases to consider writing right away, before the bug is fixed. You'll be reproducing the bug anyway (so already consider how to reproduce). You'll also have the most information about it to make sure the test is well written anyway, after building a mental model around the bug. Writing tests isn't free,…

For people who aren't getting the value of unit tests, this is my intro to the idea. You had to do some sort of testing on your code. At its core, the concept of unit testing is just, what if instead of throwing away that code, you kept it? To the extent that other concerns get in the way of the concept, like the general difficulty of testing that GUIs do what they are supposed to do, I don't blame the concept of uni…

I can't count how many times when other people ask me "how can I use this API?", I just send a test case to them. Best example you can give to someone that is never out of sync.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#147

Post by bsammon »

heikkilevanto wrote: post #36

Some additional rules: - "It is your own fault". Always suspect your code changes before anything else. It can be a compiler bug or even a hardware error, but those are very rare. - "When you find a bug, go back hunt down its family and friends". Think where else the same kind of thing could have happened, and check those. - "Optimize for the user first, the maintenance programmer second, and last if at all for the c…

Alternatively, I've found the "Maybe it's a bug. I'll try an make a test case I can report on the mailing list" approach useful at times.

Usually, in the process of reducing my error-generating code down to a simpler case, I find the bug in my logic. I've been fortunate that heisenbugs have been rare.

Once or twice, I have ended up with something to report to the devs. Generally, those were libraries (probably from sourceforge/github) with only a few hundred or less users that did not get a lot of testing.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#148

Post by astrobe_ »

heikkilevanto wrote: post #36

Some additional rules: - "It is your own fault". Always suspect your code changes before anything else. It can be a compiler bug or even a hardware error, but those are very rare. - "When you find a bug, go back hunt down its family and friends". Think where else the same kind of thing could have happened, and check those. - "Optimize for the user first, the maintenance programmer second, and last if at all for the c…

About "family and friends", a couple of times by fixing minor and a priori unrelated side issues, it revealed the bug I was after.

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#149

Post by yuliyp »

nickjj wrote: post #33

For #4 (divide and conquer), I've found `git bisect` helps a lot. If you have a known good commit and one of dozens or hundreds of commits after that is bad, this can help you identify the bad commit / code in a few steps. Here's a walk through on using it: https://nickjanetakis.com/blog/using-git-bisect-to-help-find... I jumped into a pretty big unknown code base in a live consulting call and we found the problem pr…

The principle here "bisection" is a lot more general than just "git bisect" for identifying ranges of commits. It can also be used for partitioning the space of systems. For instance, if a workflow with 10 steps is broken, can you perform some tests to confirm that 5 of the steps functioned correctly? Can you figure out that it's definitely not a hardware issue (or definitely a hardware issue) somewhere?

This is critical to apply in cases where the problem might not even be caused by a code commit in the repo you're bisecting!

Re: Debugging: Indispensable rules for finding even the most elusive problems (2004)

#150

Post by o_nate »

GuB-42 wrote: post #48

Rule 0: Don't panic Really, that's important. You need to think clearly, deadlines and angry customers are a distraction. That's also when having a good manager who can trust you is important, his job is to shield you from all that so that you can devote all of your attention to solving the problem.

A corollary to this is always have a good roll-back plan. It's much nicer to be able to roll-back to a working version and then be able to debug without the crisis-level pressure.
Post reply on HN