Giving Up on TDD

But I have seen bank code. It's terrible. I've seen credit card handling code. It's terrible too. Phones used to be reliable devices, but the average uptime of my phones has fallen to something like 3 days. Perhaps 3x/day my phone tells me that some app has stopped and that it would like to send a bug report. The code that runs cars is terrible too: http://www.safetyresearch.net/blog/articles/toyota-unintende...

Most companies I visit have bug databases with hundreds or thousands of open problems. And that's the just ones they know about, the ones where people take the time to report the bug. Things are terrible and people are just used to it. In the aerospace industry, it's called "normalization of deviance", and it's what destroy the Challenger.

None of this is necessary. It's been at least 15 years that Kent Beck has been talking about the various quality practices that drop bug rates dramatically, including test-driven development, pair programming, and continuous integration. It has been at least 10 years that Martin Fowler has been reporting on teams that have bug rates below 1 per team member per month.

I'm certainly willing to be charitable about the people. A lot of organizations making terrible software are staffed with perfectly nice people who mean well. But I decline to be charitable about the software. These are commercial products, not ash-trays made by second-graders at day camp. The software should stand and fail on its own, with no charity needed to soften the blows.

In itself this is a fair point, but I think this kind of argument only stands up for so long. The same could be said of previously relatively obscure programming styles like functional programming, but they have slowly worked their way into the mainstream as more people have learned them. The same could be said of the modern emphasis on DevOps, but again knowledge and tooling for that have evolved rapidly and gained widespread acceptance in an industry where they were mostly alien just a few years ago.

Consider for contrast Fagan-style code inspections. These too boast studies with ~90% bug yields. I don't see many people doing them.

Fagan-style is too heavyweight to be practical in most software development organisations, and rightly meets resistance as such. However, this is an area where I have considerable personal experience, and I can tell you there are a lot of places that have successfully implemented lighter weight code reviews and/or broader technical reviews of project assets, with very favourable results. Even major Open Source projects typically have some level of mandatory review and often super-review today before new code is allowed into the master branch. Almost every project that is serious about software quality has at least some form of code review process today.

Or formal methods. Again, claims of remarkable bug prevention outcomes on very challenging projects, for long spans of time. Yet it hasn't swept the industry.

Formal methods are too expensive for most projects with today's techniques. They have their place, and they can achieve excellent results in the right context. I'm bullish about the future of this field, not because I expect it to take over completely any time soon, but because I expect that some of its ideas will drift into the mainstream and become common practice as they become incorporated into our languages and tools, just as today strong, static type systems can eliminate entire classes of programmer error that are possible in more dynamic environments. However, for now the cost of heavyweight formal methods is so high that you really are into the territory where alternative engineering solutions involving completely redundant systems and the like can actually be more cost-effective.

I was struck by your point that people only ever cite the one paper. So I began looking for reviews.

I've only read one of those (the Munir one) but I'm afraid you might be disappointed. For example, of the 41 primary (mostly) sources they considered, just 9 were in their high rigour and high relevance quadrant. Of those, they report that 7 did conclude that the external quality of the TDD-based development was significantly better (one of the 7 being the Nagappan paper).

However, when you look at the primary sources, you find that like Nagappan, often what they were looking at wasn't really TDD either. For example, one was actually about moving away from TDD at a class/method level and more towards testing at a higher level with components, and it was the latter that gave the better results.

I might also challenge the classification of some of those papers as being rigorous and relevant. For example, one of the key metrics used in the Slyngstad case study is defects per SLOC, which in itself is questionable. The case study compared several releases of the same project, between which the number of SLOC varied widely (notably changing quite dramatically at the same release the TDD was introduced) but in all cases was quite small by professional development standards (only a few thousand lines). And then the paper does some extremely dubious arithmetic to reach its headline statistic of TDD reducing the mean defect density by around 35%, glossing over things like a sharp rise in the defect density in the release when TDD was introduced and the fact that the average for the test-last releases was completely dominated by a much worse score for the very first release.

In at least one case, the Siniaalto paper, the survey appears to have almost completely reversed the position of the original paper, perhaps as a result of scanning for key words and phrases a little too loosely and failing to notice that the paper was actually challenging disputing some of those claims rather than supporting them.

Overall, it's still much the same story here: some of the generalisations being presented in the summaries aren't necessarily supported by the primary data when you look at the details. There are lots of examples of the understandable but still real distortions that these kinds of surveys always seem to show up.

So while I appreciate the interesting discussion, I'm afraid we might still have to agree to disagree on this one. I'm not saying TDD doesn't or can't work for the right team in the right context, but the idea that it is innately superior to other development methods in general and the evidence typically cited to support such a claim just don't stand up to scrutiny.

To some extent, I think it is. More specifically, I think there is a balance between spending more on preventing defects up-front and not needing to spend as much on dealing with those defects later, which dominates the issue up to a certain point, and then beyond that point you have to start considering external costs as the dominant factor.

If you have a project that is made of poorly designed spaghetti, doesn't have any sort of serious test or review processes, and is kept afloat by little more than a few hero developers, then of course you're likely to have a relatively high level of defects. Even modest improvements in the development process will likely have a very good ROI in this case. In this sort of scenario I would agree with you that improving quality may be substantially cheaper than neglecting it, because relatively easy changes in development process would probably pay for themselves in reduced maintenance costs even before considering external factors.

However, the kinds of changes that bring really dramatic improvements in quality -- the kind of thing we might hope you would use for a medical device or safety-critical transport control system -- really can significantly increase development costs. Assuming you could fix the easy problems in other ways, you're probably chasing a relatively small number of extra defects already by the time you get to using these methods. To get a big jump in quality at this point, you might need to employ very different development and/or engineering techniques, such as formal verification stages, redundant systems, or much more structured and demanding review processes, and you might need to do this all the way down your tool chain in both software and hardware terms. These measures tend to require more skills, time and/or resources, and all of those are expensive.

Now, we have to be clear on what we mean by "more expensive". So far, I've mainly been talking about the development costs here, what it takes to write and maintain the software. The point of the extreme quality approaches is usually that failures of the system may have some other cost -- in human life, perhaps, or in delaying something important by a very long time -- that is not acceptable, and so extra investment in avoiding that external cost may be justified even though it makes the development itself much more expensive.

In my experience, the development costs associated with those more extreme approaches ("extreme" is a somewhat loaded term, but I can't immediately think of a better word and I hope you understand what I mean) will be prohibitive today for non-critical software, the kind of system that doesn't have a catastrophic failure case with disproportionate external costs to consider. This is what I mean when I say the market won't pay for something qualitatively better: most people won't prefer to pay $20,000 for a word processor that essentially never crashes or corrupts data or has minor incompatibilities when loading files created using its previous version, instead of $200 for a word processor that basically does its job but might crash out every couple of months and lose the five minutes of work done since the last auto-save.

By "bad" I mean "well below what teams could achieve if they applied best practices".

OK, so if we also restrict "best practices" to "things that improve quality at any cost" then I would agree with you that most software is bad by your definition.

However, if best practices also include things like being commercially viable, then I would no longer agree with the claim that most software is bad by your definition. There certainly is plenty of bad software around, but there's also plenty of software developed in ways that already do avoid silly defects reasonably successfully. In the latter case, I come back to my argument above: because of both diminishing returns in the number of failures you might prevent and the need for more fundamental changes in the development strategy that are relatively expensive to implement if you want to significantly reduce the number of remaining defects, most projects won't be able to do these things with the tools and techniques we have available today and still remain commercially viable. I don't think it's really fair to say those projects aren't well-run just because they went with a strategy that the market would accept.

And how much time have you spent practicing TDD? Have you worked on a project with 95%+ unit test coverage? Have you worked on with a comprehensive test suite that runs in under 30 seconds? Have you worked in a team that practices pair programming and collective code ownership? Have you worked on a team that does continuous deployment with at least one deployment per developer per day? Have you worked on any team that has bug rates below one per developer per month?

Other people have had different experiences than you. I am one of them. I'm telling you that it's perfectly possible to do an order of magnitude better on bug rates than most teams and get a cost decrease. Plenty of other people will tell you the same. People having been writing about their experiences like this for 15 years.

At this point, I have given up expecting J Random Commenter to believe me; normalization of deviance means that most people cannot (or will not, I can't tell which) even conceive that things could be better. It's the same way that American car companies literally could not understand how Japanese manufacturers were producing radically better products at substantially lower costs. They still generally can't, because to do so would mean admitting that they've been screwing up for decades.

So if you'd like your current limitations, carry on arguing for them. But if you would like to see if something can be different, try out something like Extreme Programming.