Why Most Unit Testing Is Waste [pdf]

No-one is arguing against automated testing - in fact, I have said elsewhere that I think test automation is the best single thing most development organizations could do to improve their process. In the article, Coplien says that his position is often mistaken for an attack on test automation, which it is not - and it is certainly not an argument against testing, either; it is an argument about how to test effectively.

What I am doing here is using the idea of automatic unit test generation to show that Coplien's argument cannot be dismissed with the claim that all unit testing adds value.

In my opinion, the way that most unit tests are written is mostly a waste of effort, but there are a class of unit tests that are much better suited to the task. The class of unit tests I'm referring to is property-based tests.

The most famous property-based testing framework is QuickCheck, but there are plenty of others, e.g. FsCheck for C#/F#, ScalaCheck for Scala, Hypothesis for Python, etc...

The general idea being that you write out a specification that the code should meet, and the test parameters are determined at runtime, rather than being hardcoded. There's more to it than that, but that's the general idea. If you want a simple introduction, I'd recommend this video:

https://www.youtube.com/watch?v=AfaNEebCDos

> In fact, we use computers for their "side-effects".

We use computers for their effects. "Side-effects" are things that happen by accident, or incidentally, when we were trying to do something else.

> I appreciate your answer, but the destructiveness of the reward loop is directly addressed in the PDF.

I've gone back through most of the PDF, and I can't figure out what section you're referring to. There's a bunch of discussion of perverse incentives (mostly involved incompetent managers or painfully sloppy developers, who will fail with any methodology), but I don't see where the author addresses what I'm talking about.

Specifically, I'm talking about the hour-to-hour process of implementing complex features, and how tests can "lead" you through the implementation process. It's possible to ride that red-green-refactor loop for hours, deep in the zone. If a colleague interrupts me, no problem, I have a red test case waiting on my monitor as soon as I look back.

This "loop" is hard to teach, and it requires both skill and judgment. I've had mixed results teaching it to junior developers—some of them suddenly become massively productive, and others get lost writing reams of lousy tests. It certainly won't turn a terrible programmer into a good one.

If anything, the biggest drawback of this process is that it can suck me in for endless productive hours, keeping me going long after I should have taken a break. It's too much of a good thing. Sure, I've written some of the most beautiful and best-designed code in my life inside that loop. But I've also let it push me deep into "brain fry".

> No amount of TDD will solve you getting half-way through a design and then finding you needed many-many because you misunderstood requirements.

I've occasionally been lucky enough to work on projects where all the requirements could be known in advance. These tend to be either very short consulting projects, or things like "implement a compiler for language X."

But usually I work with startups and smaller companies. There's a constant process of discovery—nobody knows the right answer up front, because we have to invent it, in cooperation with paying customers. The idea that I can go ask a bunch of people what I need to build in 6 months, then spend weeks designing it, and months implementing it, is totally alien. We'd be out of business if it took us that long to try an idea for our customers.

It's possible to build good software under these conditions, with fairly clean code. But it takes skilled programmers with taste, competent management and a good process.

Testing (unit, integration, etc.) is a potentially useful part of that process, allowing you to adapt to changing requirements while minimizing regressions.

There are a number of points here that I do not find convincing - for example:

The combinatorial complexity argument does apply more (much more, in many cases) to unit testing than to testing at higher level of abstractions, broadly for the same reasons that abstraction ameliorates the design-complexity problem. Consider a program using heuristic methods to find a near-optimal solution to an NP problem: not only would testing that it produces valid solutions at the program level be a lot simpler than unit-testing all its components, no amount of the latter would establish that any of its solutions are valid. The point is not that integration testing allows you to cover the whole state-space, but that it is a more effective way to select what you test.

The point of assertions is that they are tested in the execution of integration tests.

It has been my experience that most of the effort expended in the debug-fix cycle is a result of errors in decomposing the problem (or, equivalently, composing the units) - for example, an incorrect assumption about the data a unit uses, rather than the handling of the data in accordance with the assumptions made about it. Discrepancies between what is expected of a component and what it does also seem to be common, such as when one component is predicated on a constraint being respected by another, but the latter does not always do so.