Why don’t software development methodologies work?

Nicely said.

One thing I found that helps a lot is to work on a product that includes EEs and MEs, e.g. not pure software. It really drove home two points:

- process can work if people actually follow it

- the craft part of engineering takes a long time to learn but software is such a young field that practitioners are distracted by shiny objects instead of focusing on learning their craft

A couple of observations. First, we tend to push to maximize potential output. That means demanding more than could actually ever be done. I've come to the conclusion that if you're writing code that meets all requirements, on time and under budget, you're simply not ambitious enough. Pristine, perfect code is a sign of customer laziness.

The software we create today is tremendously more complex than the software we created back when I started in the 1990s. Part of that is standing on the shoulders of giants (and the libraries of giants), but part of that is also process.

The amount of process required for clear, effective communication increases with the number of parties involved. Creating good interfaces between components and the teams responsible for them is really a very difficult problem. If you're working on a three person team, it's easy. If there are 30, it's much harder. If it's 100, you probably can't even know every single person involved, much less avoid stepping on each other, not without a lot of process.

So what process does is increase the potential scope of a problem above what a small team can do (mostly by breaking it down into several interacting subprojects). That's difficult, important work, even if it's outside the realm of the average HN startup's imaginations and experience.

Legendary epic fails would seem to abolish the concept of the "repeatable process".

Imagine how silly novel authors, or fine arts painters, or technical paper authors, or poets, would look if they spent lots of time debating the "one" "true" way to produce.

A good interview question isn't how would you install the cylinder head bolts on the millionth model T engine quicker, its more like how would you paint the Sistine Chapel quicker? Are you changing the world with code or just shoveling out stereotypical CRUD app number 32515?

I agree with your fundamental underlying opinion, but unfortunately it can't be reduce to a "programmer"'s problem. More often than not, it is to the whole organization you have to show "something" that is horribly coded but will convince someone from upper management than it is doable in a few months. And then you'll spend some years debugging the horrible mess you have created in the first place just to "show something". Pretty much anyone agree this is bad, but you can't sell a project by showing test coverage statistics.

Snarky, but frankly, there will be meetings very regularly with people who care & fund the project, as well as meetings within the engineering team to keep everyone abreast of what's going on.

The trick is not to have pointless meetings. Someone should get something out of the meeting besides warm fuzzies and new doodles.