Superior Testing: Stop Stopping

I didn't have an off-the-cuff reply to this, but thinking about it afterwards the issue became clear to me: customers don't pay for code either! Customers pay for solutions to their problems.

I think the key phrase is "how do we know?": maybe those hundreds of thousands of lines of PHP were needed to solve the customers' problems, but how do we know? After all of that work, does it actually do what's needed? When the requirements inevitably change, how do we know when we've finished our patches?

We can never know for sure, but there are ways to gain confidence in what we've done. Automated testing is a really low-cost way to gain quite a lot of confidence, which is also relatively stable over time. In contrast, manual tests are either very expensive or woefully shallow; and re-running them in the future takes just as much effort each time. Static analysis, code read-throughs, formal verification, etc. can give us more confidence than tests, but at a much higher cost. Simplifying the codebase can also help (code is a liability, not an asset!), but again that can be expensive.

We should get the most bang for our buck: usually that means adding more tests. Occasionally, if that's not enough, we might sit down and prove something, or manually step though a print out, etc. but usually we could get more out of the same time by writing tests.

I think there's a direct relationship between _good_ tests and _good_ codebases. I have to say _good_ tests, because I've seen people attack in-cohesive code not by making it more cohesive, but by writing in-cohesive tests. Any test IS not better than NO tests. Slow tests are awful. Tests that are flaky are awful. And tests that don't really test properly (or miss out important edge cases) can give a false sense of safety. I've heard people say, if tests are so hard to get right, maybe they aren't the right tool. I've since come to the conclusion that: no one said programming properly had to be easy.

What I've found is that, if you test for a long time, the value of tests as a design tool diminishes. Because you gradually write code that's easier to test (and easier to test is, for all intents and purpose, always easier to read, reason about and maintain). But, you get better at testing and spend less time struggling to write tests (because the code is cleaner) so you gain less, but it costs you less.

And then you still gain the other benefits. Protection against regression, correctness, documentation (especially of edge-cases). More efficient onboarding.

Also, there's this common fallacy of: speed, cost, quality: pick two. The reality is a lot more shades of gray, but if you were to generalize it, I'd go the other extreme and say that you can't have low cost WITHOUT high quality. Cost and quality aren't opposing forces of each other, they are opposing forces of the skill of your team.

HOW you solve their problems is entirely up to you, and some solutions will work better than others.

The only thing about customers guaranteed to be consistent is that they will request changes. Some will be good changes. Some will be bad changes. Some will be unavoidable changes. Your job here is to guide them through the best possible change process, with the ultimate goal of solving their problems.

Testing is a useful tool for (1) making sure the system matches the customer's expectations, and (2) making sure new changes don't break old things.

How you implement tests is entirely up to you, but once again, some solutions work better than others depending on the situation. Manual testing only scales up to a point before the cost of the manual testing is greater than the cost of implementing and maintaining automated tests. You need to become good at estimation here, but for 90% of projects, automated testing is the most efficient long term strategy.

WHERE you test is also important. Generally, it's best to test components at the edge of their interfaces. When you input X, Y should come out, ideally every time. This is where immutable data and idempotent APIs are VERY helpful for maintaining a reliable codebase. The more you need to cut into a component's innards to test it, the more you should be asking yourself why.

Testing is very much about architecture. Make clear boundaries between components and outside interfaces, use immutable data and idempotent APIs, and tests become a lot easier to write, and more resilient to change.

At my current work we have loads of automatic tests. Every time something is changed, there's pipelines that test the following:

- Simple unit tests: Start with state 0, make and action, is state 1 what you expect? There's literally hundreds of these, and they have indeed found issues. Many edits that would have gone in have had to be rewritten to cater for corner cases found in the tests.

- Valgrind suite tests: Since we're dealing with c++, you get some interesting errors that might happen. Changing the order of some lines can look innocuous, but sometimes that causes an iterator invalidation. This kind of thing is hard to spot, but we have automated tests that will stop you merging code that does this. Especially things that are UB in c++ can be very tricky.

- Integration tests: several pieces that are in themselves passing unit tests might not work together. Luckily you can define a CI script that launches both and makes them talk to each other.

As you can imagine, it takes a lot of work to write the tests. One issue with a lot of teams is there simply isn't enough time. You have to show progress, which means showing things that seem to function externally. But you're always paying for that by having to fix things that you find as you go. Not having tests is a form of tech debt. It costs more and more as your code base grows.

I forgot to mention this part. We have data recorded from external sources, which we run through our code. Millions and millions of state changes, along with their summary states. This also helps to find cases we didn't think of.