Why Can't Developers Estimate Time?

Good post Ash (good to see the discussion being consolidated)

IMHO - it's usually due to the lack of "up front planning" by Managers / Analysts.

If you compare building software to more general "building endeavors" (for example building the Olympic stadium) then, IMHO, developers are really the "construction workers" (the people actually doing the work). However no intelligent Olympic committee would countenance starting work without (for example) an architect making an incredibly detailed plan (and even a working replica model).

IMHO the problem we suffer from as developers is most often diving in too fast because: a) we're the construction workers b) when we do use Architects they are "construction workers too" (as opposed to being a trained with 'different but complimentary skills - it would be hard to imagine Sir Norman Foster laying bricks)

May be because software development is a creative process or the same reason why Artists won't tell you how long they will take to finish a painting or Authors repeatedly miss publisher deadline for books.

I estimate by considering an optimistic, pessimistic and normal value for each task. I then have a spreadsheet that does a modicum of statistical analysis to give you times against probability that the task will be finished.

Some of the benefits I find are:

1. Getting into the pessimistic mindset helps produce better estimates

2. Tasks with a large spread are things you aren't really clear about, and are an obvious target to be attempted first

3. Giving a range of values to the person asking for the estimate seems to help them remember it is an estimate.

There are human timescale factors involved.

If you give an estimate that's too far out, it'll be outside your client's planning scale. That means it's so far out, she doesn't know what will happen by then, she won't do the project. In my experience, this is scale is roughly 9-12 months, unless you're dealing with a MegaCorp.

So most estimates end up at 6-9 months, which then slip into 9-12 territory, and hopefully not much further =)

In my experience, coding time is usually easy to estimate, and that's what most estimates coming from developers should be taken to estimate. But then you have integration (nothing works), testing (everything is broken), and then realizing what the client meant when she said X (solved the wrong problem, optimized for wrong features), those are the tough ones.

You should take the estimate for coding time (coming from the developer), and multiply that at least by 2x. Of course, going back to my original point, that estimate may be outside the "human timescale", so you may not be able to tell the client that...

So, to answer your original question, developers can estimate primary development time up-front pretty well, but they suck at estimating integration, testing and shipping time.

A related thing I noticed is that if you ask a developer during the late(r) stages of the project when she'll be finished with the "next stable version", the answer usually hovers around "in 2 weeks", even if the actual answer is more like 3 months. The developer isn't lying, she genuinely believes that it can be done in 2 weeks.

That's why you want experienced engineers and not fresh college grads leading teams and giving estimates. They've gone through these experiences, have a lot of soft data metrics in their heads, and can come up with good multiplicative factors. I'd be curious if orgs like Google have a database of project data to aid planning.

Because developers don't work in a vacuum. The second you estimate the project, the client starts changing it...