Why don’t software development methodologies work?

An important realization I made a while back was that design methodologies do little to address program correctness, which is almost always the wildcard on deliverables; buggy software means missed deadlines and budget. Some, such as TDD, work to address the rapid building of tools to a particular spec, but often fail to promote static guarantees, especially in languages and environments where such provability is largely impossible. Dynamic languages penchant for monkey (guerrilla) patching further exacerbates the problem.

Solutions to this are tough. My first suggestion would be to use languages which facilitate correctness, although it's usually at the expense of developer availability: the pool of engineers with experience and know-how in true FP is orders of magnitude smaller than more pervasive languages. My second thought is to further embrace math as the building block for non-trivial applications: mathematical proofs have real, quantifiable value in correctness. I find it no surprise that the larger companies have made foundational maths, such as category theory and abstract algebra, the underlying abstraction for their general frameworks. This is even a tougher pitch than the first since most engineers don't recognize what they're doing as math at all - a big part of the problem. So many of us are doing by feel what has already been formally codified in other disciplines.

I'm aware that both require more (not necessarily formal) education than most engineers have pursued and makes it a difficult short-term pitch point for any company, but I think if we're serious about eliminating sources non-determinism from projects, it's important we address them directly.

> rigorous studies of software development methodologies haven’t been done

A simple search proves this is false. Paired with author's other unsupported and stupendous claims I have to doubt the validity and worth of anything else author says.

I think there's a difference between having an initial set of requirements, and believing that those requirements are the requirements forever and ever, amen. Waterfall relies on the latter, while Agile assumes that they can and will change.

Both projects will be given the set of initial requirements. After all, if you don't have requirements, what the hell are you even doing? It's what happens when those requirements change that's the fun part.

This would rather much be like a carpentry shop owner lamenting "why don't either electric circular saws OR handsaws work to make my mediocre workforce produce beautiful furniture without all of this wasted wood?"

First of all, creating beautiful furniture takes LOTS of time, as it rests in attention to detail and an uncompromising position towards quality. When you're uncompromising about quality, you require yourself to throw out, to waste, things that are of lesser quality.

Second of all, beginner, poorly trained, and dispassionate employees are never going to produce beautiful furniture. Mostly because of the above: they either lack the ability or the caring to have the attention to detail necessary to do great work.

And the best carpenters aren't going to come work for you unless you're willing to make it worth their time. Coming to work for you means they will have to do a lot more work in a much less comfortable environment than they are used to. Because the best carpenters have their own shop, their own tools, and work on their own time, because they love it. Going anywhere that is not their own setup is automatically worse for them.

It's the same thing in software: great software takes patience, it takes time, and it takes money to convince the best programmers that they should be spending their time on you rather than on themselves.

But at no point does any of that mean that, because Tool X can't magically turn your mediocre working into a stellar one, that means that Tool X is not worth studying. That is completely, 100% backwards. The master practitioner studies all tools, even the ones he or she doesn't like, at the very least to understand what is wrong with them and why they don't produce the results they want. A master carpenter might prefer a hand saw over an electric one because the electric saw chips the corners of the board too easily, or something. But he doesn't know that unless he's studied the electric saw.

You can't have "an absence of methodology" any more than a wood shop can have "an absence of tools". It has to be there. But they all have their pros and cons, and non of them has a pro that includes "makes men out of mice."

Because Software is inherently complex, one of the most complex things that Humans can try and make. And whenever we successfully make something easier, we push the borders out so that Software is always stretching to the limits of the complexity we can handle. Ultimately, its just hard. Sometimes methodologies help, but there is No Silver Bullet.