Why Development Teams Struggle to Deliver on Time, on Budget, or at All

It's called spec programming, where you have programmers slinging code against precise specifications. Common in ERP and regulatory or business compliance-driven code.

On the other hand, you have development. That is where you're attacking the less defined problems. Sure, we know the tools in our toolset, how to apply them, and time it takes to apply each tool. Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time, time that could have been spent towards a MVP.

In the use of the adjectives 'precise' and 'enormous', you are just repeating the same false dichotomy in a slightly different form, by ignoring (or not understanding) my point about the use of abstraction in analytical thinking.

The people who are most successful in what you distinguish as development understand a good deal more about what they are working on than spec programmers need to, and they are not, as you imply, groping around in the dark for a solution - that's what spec programmers do when faced with a development problem.

The first involved taking all sorts of situational data from various sources in an airport, e.g. radar systems with their own binary formats, and combining them with the work of a postgrad researcher who had come up with an algorithm to optimise when pilots should turn on an aircraft's engine in order to save fuel and cut emissions.

The second involved managing 3G dongles on fleets of laptops, tracking and sending alerts on their usage and billing info in realtime, back in the days when business users would accidentally run up £10,000 bills abroad.

If at any point in the first half of either of these projects I'd been asked to come up with an estimate for the total amount of work, it would have been wildly inaccurate. Each project had both low and high level surprises at every turn that would have just compounded to blow those estimates up. There is basically no significant way in which one informed the other, even though parts of the systems used similar event sourcing architectures on the exact same server-side stack, and were done for the exact same startup which broadly offered real-time systems to utility companies of various sorts.

It's entirely possible that there exists a breed of programmer who could have estimated these projects in their entirety, much earlier and more accurately than we did, but I've never met them, and I'd rather work in places that didn't compel people to.

I've known and worked at other places where basically any software methodology would (and did) work, but I certainly don't think it's rare (as you seem to suggest) to find yourself at the other end of the spectrum.

1. Managers (and developers, too) don't strive for repeatability and predictability of the process. Sticking to frameworks, reducing variability, forcing an exact development pipeline. It is not appealing to developers and managers are afraid to push this.

2. Lack of feedback loop. At the end of every project/deliverable/iteration, ask what could have been done to prevent the problem (what could have helped to estimate this more reliably). Implement mercilessly as if bad estimate was on par with deployment failure.

3. Move unplanned work to planned work. Prioritize delivering sound and good 100% of code over delivering quickly "the first" 80% of your solution. Develop code as if it was controlling Solid Rocket Boosters. Follow good practices like MISRA. Don't allow exceptions. Do PROPER code reviews. Most code reviews I have seen is a colleague spending minimum possible time so that he does not feel he did bad job. In my opinion good code review requires going throug evereything top down from requirements and then bottom up through each statement to understand everything is implemented correctly. This takes about as much time as implementing it in the first place. Make sure managers understand this typically takes more time to do correctly and they should expect payout later, not this iteration.

4. Hire carpenters and make them your senior staff. Make sure you have clear understanding who is senior/junior developer. Senior developer is a person that understands broader context and can be left to supervise a small project with understanding he/she will be able to uphold standards and provide correct solution and guidance for the junior staff. Make sure your senior developers are carpenters -- most projects don't require exceptional skills and rock star developers. They require people who don't get bored once they see something working but instead have the drive to finish the second 80% of your functionality and do it with the same amount of focus as when they have started.