Alan Cooper on the “10x” Programmer

Bumping the test coverage up past 90% definitely has lines re-run multiple times, with core logic even being re-executed possibly hundreds of times. This is absolutely ok! Your main purpose of writing tests is to make sure when you change business logic that a test will point you to the exact spot. If you commented out any single line of business logic; a test should fail.

I understand that this may seem crazy if you are used to 80% coverage, however there are subtle ways to build your unit/integration/feature/security/performance tests where you only pin your public api, and code changes to the internals don't break the tests, and the test suites only have minimal (necessary) overlap. This is test engineering and it takes about a decade of TDD to be comfortable with.

I would have said 100% test coverage for 100x but that last 1% isn't worth it. Some things just can't be tested without altering the public interface of your code JUST to accommodate the test, or they require some crazy significant re-work of the code that makes it hard to follow, or you have a module with a crazy way of interfacing with it that makes testing a specific part extremely difficult, or it's code that doesn't really need a test (like a log statement).

Side note: 99% test coverage isn't typical but is absolutely necessary if you have automatic patch-management solutions in place and are striving towards zero-defect code. How comfortable would you be pushing to production after bumping the major version of your platform and seeing your test suite pass? Having 99% confidence in this situation is a good thing and saves you time in the long run.

10x programmers can hit a target no one else can see. This is true for 10x persons of all types in all endeavours, not just programmers. In this mode they are more artists than technicians. Compelled by vision from an overflowing sacred gift outpacing its faithful servant. Conventions and norms are not designed for this. Those who push beyond the bounds by definition cannot be bound by them. If you want to maximize their productivity, provide them with supporting infrastructure and surround them with brilliant editors to optimize this process and integrate their vision.

I begin to relax around 85%.

It’s perfectly possible for a project to hit 85% without much untoward going on. Running the code and testing the code are not the same, and as the numbers go up I tend to find shenanigans.

I’d rather have an honest 85% than any dishonest number you could want. My butt starts to pucker again above 95%, because it’s all lies.

There also comes a point where running certain tests by hand is faster or more reliable than having a test for it. False positives (failing tests) are expensive, and in a way that is often counted poorly.

For a start, a false positive doesn’t cost the time of the test, it costs the time of the entire test suite times three - the one you wasted, the one you do over, and time between getting a failure and noticing the failure.

It also costs trust in the tools. Red tests become untrusted, which really hurts when you have a test that has false negatives. People become conditioned to hit the run button again without understanding the error, which then blows up farther down the line, possibly in production.

Testing is hard. Most of the people who tell you otherwise are full of... bravado. Sometimes they have the worst tests.

When I started working with more dynamic languages I took comfort in the promises from TDD folks that you can write tests to cover all of these things, but I've seen how tests are written in practice and it's not good. I've met maybe ten people in twenty years who are really competent at writing bulletproof tests (as in, "You wrote tests for that? Enough said" versus Trust but Verify) and I am not at the top of that list. Frequently enough, I find eyerollers that have my name by them in the commit history.

I'm not saying I'm ready to go back to static languages because of this, but I will say that I see evidence for hope on the static analysis side for once in a very long time.

As you say, the problem is often discerning between unit tests and functional tests. Some people clearly can't tell the difference, but a number of people have confessed discomfort at writing unit tests (something about my frank communication style seems to invite people to share their fears with me, and I hear things that the "everything is great" people don't due to self-censorship). That these unit tests seem juvenile, pedestrian. Remedial. But to my mind that's the point. The cognitive load of determining why I just caused a bunch of tests to go red should be near zero, since I'm still trying to keep my head wrapped around the new code I'm in the midst of writing.

For about as long as I've been writing tests I've worked with people who think writing their own mocks or helper functions in tests is a good use of time. They take some work, so once you have them working they write five more tests that use them. The poor SOB who gets a red test six months from now has no idea what's going on.

Worse, adding more to these mocks may cause some of the tests to be evergreen, and others to be required (deleting test 1 makes test 3 fail). Which is only discovered when a customer complains about bugs and you say, "wait, don't we have tests for this?" The tests are coupled which is bad juju. Every test should be capable of being run in isolation (debugging your tests!)

Often the first thing I do is rip this mess out and winnow the mocks down to the one, two, or three calls that actually need, so each test can be taken at face value. I also replace a lot of ten line tests with more four line, transitive tests (if A->B and B->C, then A->C.) Then I remove any coupling between the input and output that wasn't fixed by doing that (once a month almost, I find a test whose assertion boils down to expect(a).toEqual(a)). And the utility functions get split up into matchers with much more diagnostic data added in, and a bunch of incidentals taken out.

Effectively what we are doing versus languages where writing the code up front requires more thought, is that we are marking stories as done that aren't properly done for weeks, months, or years. It's a bit of knowledge that makes me uncomfortable. I have a lot easier time hiding that discomfort with people whose position in the org chart implies that they are still learning, and quite a time being constructive to the braggadocios. It's a lot of emotional labor and it's taxing.