DiffDebugging

DiffDebugging
martinfowler.com

Let's say you've been working on an "algorithm", which is a combination of cleverly-crafted SQL to fetch the right data, some ad-hoc data processing code in Python, getting predictions from a model, and some additional mathy stuff to translate the model prediction into something useful. You probably built it interactively in a REPL or notebook environment. You might have some ad-hoc assertions scattered across your scripts, but nothing resembling a test suite.

That's just how things go, and if you did it differently you'd probably never get your work done. But now you have a problem: you have what you believe is a mostly-working/mostly-correct implementation, which you built incrementally and interactively, but now needs to be put into production. Without any real tests, your script has immediately become "legacy code": https://understandlegacycode.com/blog/what-is-legacy-code-is.... If things went well, you had time to extract some common utility functions and write unit tests for them, but you probably don't have more than that.

So now what do you do? You want to refactor it into testable units, so you can build out a proper test suite. But you can't refactor code without tests, lest you risk breaking something. You did extensive manual checking and validation of your outputs, but you can't keep doing that over and over.

In this case, the best option that I've found is to do pretty much what Fowler is advocating for here. Extract a set of known inputs that don't take too long to run through the algorithm, and save their known-valid (or known-close-enough-to-valid) outputs. Then, piece by piece, start refactoring, re-running at each stage. If the outputs differ by more than floating-point roundoff error, you've either found a bug in your original implementation, or you introduced a bug in your refactor. Once you're sure that your refactor is OK, you can "lock in" the changes by adding new tests for the refactored sections. Repeat until satisfied, or out of time.

It's also possible to do this is a Bayesian fashion.

I can't speak to 2004, but these days "bisect" is the standard term in my circles (even when we're not actually using git and its bisect command).

It would be interesting to know whether git was the first to associate that word with this activity... especially given it's atrocious naming of everything else :P

Bisecting is useful when you find a bug which your test suite did not, and which may have been introduced a long while ago. So you search for the first offender between the last known-good commit and the most recent, usually with an automated test that reproduces the bug, and which should be introduced in the suite to prevent it from happening again in the future.

A snapshot test is basically a “let me know if this unit stops giving this output for this specific input”. And sometimes the fix is to acknowledge that this new output is now the correct version by overwriting it to the snapshot.