Why software projects take longer than you think: a statistical model (2019)

Start from an empty file and write to an imaginary API or pseudocode what you actually want to do and coding is easy and fast!

I think the high level imaginary API/pseudocode of any system is straightforward and fast when you start from scratch. (Because you're not focused on detail and just intent)

But when you start from an existing codebase, existing infrastructure, existing APIs, it slows to molasses. You've got to understand and have a mental model of ALL OF THAT before you can complete your intent.

I think this is why software takes longer than expected. Your mental model has to be accurate, and there is always more to learn.

How do you solve the problem?

I'm an advocate and trying to design what I call "commutative programming", which is another way of saying that the behaviour is the product of every statement about the behaviour that is desired, not an explicit instruction of what to do next which is what modern programming is and it is slow and tedious and doesn't compose. In this dream, if you want to change behaviour, you just add statements. Programming is additive! Not edit sourcecode or try to find out where to insert your lines of code or tweak existing lines. Let the computer tell you about conflicting requirements or missing criteria.

We need query engines and cost based optimisers for behaviour.

Imagine taking lots of files that are simple "writing to an imaginary API" - that's your intent specification. You just need to merge them together and your system is finished.

I started work on a commutative GUI https://github.com/samsquire/additive-guis but I'm also thinking from time to time about commutative code where we define refinements to desired behaviour and the computer behaviour query engine or rule engine generates the code or configuration that exhibits this behaviour. (A bit like Kubernetes for behaviour.)

In the future: we don't get LLMs to code for us, but we have highly expressive semantic vector spaces that map to architectural relationships and meaning of behaviour, and changing the architecture of a system to support new behaviour is a matter of changing the serialisation of semantic vectors via the prompt.

This model is working to identify the Inspection Paradox [0] - most time is spent on doing tasks that take a long time to do. Combine that with high variance, and one task will blow out and a project will be spent working on something that is much harder than people thought, and that the estimate process didn't see coming.

My interpretation here is software engineers are better at estimating tasks than I thought. The next step is analysing tasks being performed in serial to show how many sub-tasks a project can have before the estimates are effectively meaningless / likely to blow out an unpredictable amount. I bet it turns out to be about a sprint's worth of work. Real people have a knack for feeling out those sort of tipping points empirically.

[0] https://en.wikipedia.org/wiki/Inspection_paradox#Inspection_...

I think that's a pretty smart model - best/worst/most likely. One thing I found lacking in kabuki agile theatre was that we were ABSOLUTELY NOT supposed to "get into the weeds" in meetings where we talked estimates.

But how does one uncover the best/worst/most likely without doing so?

You can spend 20min writing the most beautiful Acceptance Criteria but if you don't get a bit into detail on what the user actually wants it's all for naught. It could be way simpler (hey what if I just did X) or way more complex (oh well now that you mention it..), etc.

Typically it was engineering manager & engineers sitting in a room discussing requirements with each other. What could go wrong obviously? lol.

What I find ironic is that everyone in the industry knows that estimating is very hard, thus estimations are nearly useless, yet everyone insists on the importance of planning and on basing such planning on void estimates.

Nobody has yet had the balls to state the obvious: we have to learn to work without estimations.

> no-jira, no-meeting, get-it-done environment

Best env, but sadly a bit rare?

I used to select for shops like that, no wonder lay-offs are scary if you've been trapped in meetings instead of shipping.