Why Can't Developers Estimate Time?

I've spent my career as a lawyer dealing with this problem.

The solution is to choose a price that represents the value proposition of the output (whether it's code or contracts). As part of that price, you will bring a comprehensive process to the table – one that is very clear about what the client is expected to provide.

I have gathered a huge amount of timekeeping data from lawyers, and crunched it up, down, left, right and center over the years. Here is what I learned:

1. If a task can be defined, it has a predictable price.

2. Tasks should not be measured by how long they take start to finish, but rather by how much time is actually required to complete them.

3. A clearly defined task always takes just about the same amount of time. The start to finish time is variable because:

(a) the client is faster or slower at providing resources or information; and

(b) your other client work is causing bottlenecks in your own availability.

4. You can solve for (a) and (b) above. I'll leave it to your imagination how. That's my secret. ;)

Of course, most development projects are not like that. The pure coding part of it is interspersed with countless decisions about how the app should work, with each decision requiring input from lots of people, plus a bit of experimentation and rework. The process of getting to a decision is hard to estimate and the amount of development that each decision will create is impossible to know until the decision has been made.

Estimating a software project is like estimating the length of time it takes to get a law passed. It's not a computer science or even an engineering matter, its a political/ psychological one.

Really, it's easy for developers to estimate time much the same way plumbers, carpenters, auto body shop workers, general contractors, dentists, etc. do: When a developer has done a very similar project more than 10 times recently, they can give a good estimate for how long it will take to do the project once more.

This situation is quite general in engineering and other work of wide variety.

Similarly, in general, when some project is being done for the first time, with no prior experience with comparable projects, then time estimates are tough to give.

In particular, for 'developers', now they work heavily with 'parts, pieces, and tools' from others, and working with these for the first time encounters standard problems: (A) How to use the parts, etc., needs good documentation rarely available. (B) Newer parts, etc. commonly have strange behavior or actual bugs that have to be encountered, diagnosed, and worked around. So a developer does not know how much time will be spent mud wrestling with (A) and (B).

This is where having good historical records is useful. If I have done 2 similar tasks before and they took a month each, then a 2-week estimate has no validity.

As far as general code estimation goes, lines of code and complexity taken together form a pretty good metric. If I estimate I have to write 50 lines of high-performance multithreading code, I might say it will take 8 hours. If it's 50 lines of fizzbuzz, estimate drops to 1 hour.

But like you say, that's just coding (and unit test). The full task probably requires (1) getting or understanding requirements, (2) at least some cursory thought about design and (2a) having a hallway review with somene if you're not sure. Then when we're done with code, unit test (don't forget to estimate how long it will take to create, debug and run the tests) and Code Review (how many reviewers do you need, how long will it take them to get around to inspecting your code, etc), someone has to estimate (4) how long the test team will take to write test cases and do the system test of the new feature, (5) tech docs has to update the user manuals, (6) sysadmin or the Release Master estimates how long deployment will take.

In short, at least in the environment I work in, that 50 lines of code that took eight hours to write, will easily require an entire week before any customer can see it. In my experience, it's often not so much that the developer didn't estimate accurately how long coding would take, it's that the developer never thought about all the ancillary tasks that have to be done (code review, unit test, documenting the design changes, etc), so they were not in the estimate.

The complexity is simply too high and it's not getting better. One of the reasons I think why you see the fail fast movement be so successful.

Once you accept that failure is part of the process, once you abandon the "zero mistake" policy that many large organizations instill internally and externally you will begin to approach projects differently.

The truth is that "zero mistake" organizations make as many mistakes as everyone else, they just have the financial strength to ignore them as long as economy of scale works in their favor.

I could write forever about projects that went wrong not because the developers where bad but because the premise that fuels product development is broken.

I blame primarily business schools and large parts of academia for this. But it could extend all the way into the way the stock market is structured.

If you buy my premise that post-industrial is different than industrial age. That project definition is primary and time is secondary today. Then it does put some doubt at least in me about whether the stock markets focus on growth and Q's is sustainable.

Nature seems to be doing a good job as pacing various processes. It takes nine months to give birth to a child. One cell at a time. But the process is ongoing.

Nature is the ultimate continues deployment strategy.

Edit: Fixed lame false modesty in the end!