Why are software development estimates regularly off by a factor of 2-3?

I enjoyed reading this the first time around. I purposely illustrates the complexity of software and perhaps inadvertently illustrates the challenge of getting schedules from inexperienced teams. In rebuttal, had they walked from San Francisco to LA along any freeway they would have been done much closer to their predicted date. Which they would have known had they been experienced in walking back and forth to LA from the Bay Area.

The "big" message is that if you have never done something before, you have no idea how long it will take, even if you can describe what you are going to do pretty easily. Returning to the hike metaphor, this is why you need to listen to the cranky old guy saying "walk along the roads, not the beach" instead of the inexperienced folks saying "Hey its just walking, and walking on the beach rocks! Walking along a road is boring and lame!"

I love this post; it conveys the feel of the experience so well.

One of my big aha moments about estimation was a bit in McConnell's Rapid Development. He pointed out that most estimates get made with executives pressuring for short numbers. When you iterate a few times with that, you end up with the smallest number that developers can't absolutely prove is impossible.

If you draw out a bell curve of probable completion dates, this is basically the same as picking one far to the left, so you have a 95-99% chance of being late. But somehow, executives are still surprised when their totally biased approach yields the numbers they liked, not the numbers they needed. All the estimation effort was pure waste; our time would have been just as well spent making cotton-candy raincoats.

I now have done a few projects with little to no estimation, and it has gone much better. I have a hard time now seeing why we ever bothered.

Management giving estimates? No thanks. I'd rather them take my input and do their job of understanding trends. When I say something takes a long time and it takes longer, I'm not bullshitting. When I think something takes a long time and doesn't, its generally because shortcuts are rarely understood up front. Will I find the same shortest path next time? If I'm doing the same exact work maybe.

I do work in a place where we are a software company but very few people, even some of the devs, understand this culture. I don't say something is complex just for the hell of it. I want to give good estimates but at the end of the day that is all it is, a somewhat educated guess before I've gotten in the weeds. I don't need people that absolutely don't understand this concept to mandate a timeline that is unrealistic. It does absolutely no one any favors. That is why I leave. I don't need to promote unrealistic expectations down the line to customers who think everything is extremely simple. Especially when all we do is create custom software where almost 0% is turn key or off the shelf. Its just insulting to continue that nonsense and the quicker I can facilitate a reality check for all parties, the better and more trust is given to my judgment (and theirs if they actually take the time to learn)

Have you shared these spreadsheets and lists online? They sound like something that could turn into an excellent, three-figure or more information project.

also because we don't have Time Machines