TDD did not live up to expectations

I'm not a TDDer but must object on their behalf: not only is that not what they said, it couldn't be further from it! What they taught was to work in short iterations and work out a bit of requirements, design, and implementation in each one. If you needed knowledge you didn't have yet, they would advise you to make each cycle shorter until you did. Sometimes they'd take this to absurdity, breaking even simple features into design-test-implement cycles so microscopic that you would feel bored like a kid in class who can already see the next dozen steps and has to sit there waiting for the teacher to catch up.

The TDD approach was at the extreme tip of trying to get away from waterfall methods, so it would be quite some irony if it reduced back to that.

Edit: looks like I misread what taeric meant! But as McLuhan said, 'you don't like those ideas, i got others'...

In my experience there were two problems with TDD. One is that the micro-iterative approach amounted to a random walk in which features agglutinated into a software blob without any organizing design. Of course that's what most software development does anyway. But if you remember what Fred Brooks et. al. taught about having a coherent design vision for software that can organize a codebase at a higher level, the way iron filings are ordered by a magnetic field, that is a key anti-complexity vector which the TDD/XP/agile projects I saw all lacked. (Original XP had a notion of 'metaphor' that pointed vaguely in this direction but no one knew what it meant so it had no effect.) The lesson was that a coherent design vision doesn't evolve out of bit pieces, and there is an important place for the kind of thinking that got dismissed as 'top-down'. (Edit: by far the deepest work on integrating design coherence into TDD/XP approaches was Eric Evans' stuff on domain models and ubiquitous language. Eric is my friend but I knew he had the best stuff on this before that :)) (Edit 2: Eric points out that the people who created TDD were software design experts for whom coherent design was as natural as breathing air. The admonition against too much design thinking in TDD was intended for people like themselves, for whom the opposite mistake was never a risk, and didn't have unintended consequences until later.)

The other problem was that practices that started out as interesting experiments ossified into dogma so damn quickly that people started acting like they'd cracked the code of how to build software. A soft zealotry emerged that was deeply out of sync with the playful, adaptive spirit that good software work needs. This was unintentional but happened anyway, and there's a power law in how much creativity it kills: each generation that learns the Process is an order of magnitude less able to think for itself within it, and so must break free. I don't think the creators of the approach were any more at fault for this than the rest of us would have been in their shoes. The lesson is that this ossification is inevitable, even if you know all about the danger and are firmly resolved to avoid it. If there is any real breakthrough to be had at the 'how to build software' level, it is probably discovering some kind of antifreeze to put in there from the beginning to make sure this doesn't happen. Maybe there's a way to interrupt the brain and social circuitry that produce it. Failing that, I think that each project or maybe (maybe!) organization should hash out its own process from first principles and the prior experience of team members. That's ad hoc and involves a lot of repeating past mistakes for oneself but at least it isn't idiotifying. It's how I like to work, anyhow, and IMO that's what all software processes reduce to.

Way back before anyone really understand how software engineering should be done, people would begin a software product by attempting to pre-specify ahead of time every nuance of the system. They'd write these massive books of requirements. Then they'd get to work coding to these requirements.

This rarely worked out well and usually resulted in huge amounts of wasted effort. When you get coding you realize the spec is either wrong or there's a much, much better way to do it. Do you do it the bad way to stick to the spec or do you deviate from the spec?

The other problem with this approach is that the requirements are often as hard to write as the code and contain as much information. Why not just write the code?

TDD is just "the spec" written in code instead of as a massive design document. Attempting to achieve 100% test coverage often means writing a test set that is a kind of negative image of the code to be written and is equal in complexity. As such the tests often end up containing bugs, which is the same problem as the classical "when we got going we found that the spec was wrong." Do you code to the bugs in the tests?

This is not to say tests should be skipped. Automated tests are great. I just think the 100% test coverage TDD dogma is oversold and doesn't really work well in practice. Test where you can. Test where you must. Keep going.

Of course I am not a fan of "methodologies" in general. I see them as attempts to replace thinking with rote formula. That categorically does not work outside domains that are purely deterministic in nature, and those can be fully automated so humans aren't needed there at all.

There is no formula, or alternately the formula is general intelligence continuously applied.

Edit:

The same evolution has happened in business. Nobody writes a "business plan" anymore. As soon as you get into business you have to toss the plan so why waste your time. Planning is a false god.

Basically, I'm not intending something different than what you are saying. I just picked a phrasing that can go both ways, unfortunately. :(