Why are software development estimates regularly off by a factor of 2-3?

Spot on. NTE billing puts the onus on you and gets you actually thinking about it. Helped me a ton.

Good points in the article.

You can think of it this way. A coast line is a fractal line. As you zoom in its length gets bigger and bigger seemingly. From the top level on the map it looks fairly straight.

Related to estimates. Programmers have and operate with idealized mental models. A FIFO queue works kind of like this ("/closes eyes and sees a line of people lining up at the store"), a tree traversal looks like this ("/closes eyes sees a red cursor move down a tree drawn with blue nodes on an white board"). Same with estimates. How long will this take? ("/closes eyes and imagines an idealized sequence of events that will lead from now till project completion -- use this library, develop that API, test a little bit here, done... 2 weeks").

The problem is, zooming in is hard. "Oh this library we are about to use actually doesn't implement these corner cases". So now we are spending 3 days patching it. Oh, while working on this project, a critical support ticket comes from a customer. And then testing reveals a fundamental flaw in our API design. Re-write a huge chunk and repeat.

Some developers and managers just use heuristics. They do the mental estimate then x2 or x3.

^ All the above is true if the developer is everyone is honest and trusts each other. As other posts wrote, that is not always the case.

Then it is a more complicated game. The main question to ask then is "What is the punishment for overestimating vs the punishment for underestimating?".

Have people been fired for underestimating? No, ok underestimate. Have people been fired for missing deadlines? Ok then overestimate.

You can even play this game with yourself. What kind of deadlines do you set for yourself and how to you handle setting a too short or too long of a deadline? Do you get better with time at setting deadlines? If no, why not?

Because software developers, like most people, don't like conflicts.

Most people would agree that practice makes perfect, right? I think a big issue with software estimation is that it is hard to get practice. I'd done full project estimates for 4-6 projects in 3 years of working professionally. Compared to the amount of practice I have writing code, this is nothing. How can I ever hope to get good at something if I only do it once every 3-6 months and it might take 2 years to get feedback on if my final number was close to the actual cost? And unlike other skills, I can't practice on my own or take a course (I've never seen an open source project that did round-trip estimates, maybe that's something to try out...)

Another issue is when you have an "indirect estimate" - basically I will estimate the work, but someone else is the one that ends up doing the project. If you aren't careful to consider who will be doing the work, you might estimate too low (if you are an expert and the work is done by a bunch of new hires).

And none of this even touches on misinterpreting client demands, scope creep, dev team turn-over, or often neglected timesinks like documentation and meetings.

This post was originally an answer on Quora to the question posed in the title [1]. The question has so far received 222 answers - some of them arguably better, though not as popular as this one.

[1] http://www.quora.com/Engineering-Management/Why-are-software...