Why are software development estimates regularly off by a factor of 2-3?

Here's my experience with this problem. First, a number of people go meet the clients, often several times, to see what the client needs. Generally however, none of the people involved with actually building the product are ever involved in the meeting (because, from the mouth of the Boss, "operations are too expensive"), and in the unlikely event that they are, they're essentially shut down because "nobody really cares about the technical details at this point"...

This is then followed by extremely loosely defined requirements being passed down to the technical people in hope of getting an estimate back. Technical people, especially experienced ones, know how this goes. First, you know you don't have half the details you should have, even though you've most likely asked for them numerous times. But you know, "all I need is an estimate" says the sales guy (who in this case also happens to share the board, made solely of other sales related people). You already know you can't win here, so you give a ballpark estimate considerably above what you imagine should be the real value, hoping it'll cover your ass, but highlighting the fact that until more details are given, this is still a ballpark figure.

It doesn't matter. The sales guy takes your estimate, shaves a third off of it, throws in a couple of bonus features in there (which are most likely of no use to the client), and makes it a hard estimate. When you point that out, you're told not to worry and that it's not your problem (spoiler alert: it very much is). So you write your specs down, or rather, you slap a few bullet point lists together with whatever little time you're given (you're expensive). This is given to the client as what both sides should abide by, despite you repeatedly saying that it's a bad idea.

From that point, you realize that the time you've been given to achieve the work is, roll drums, not nearly enough. You can already see poorly defined features growing arms and legs: clients, and I can't really blame them, will use any vague requirements to their advantage. You also realize that in order to do all the work and still meet the deadline (you still most likely won't), you have to half-ass a bunch of the work. You can already imagine the support issues pouring in once the project is in production, and you see all those legacy projects constantly getting in the way of your new, poorly estimated work, thus slowing it down even further. You will also get blamed for all these issues that are cropping up from past projects, and this ties to the next point.

The best one of all, that thing that wasn't your problem, suddenly is. You've somehow become responsible for the poor estimates you clearly said weren't to be relied on. So you know, you better make an effort and fix that by, say, working extra hours here and there, like, every single day. You burn out and grow frustrated, which slows the work down even further. Even better is that your crazy work hours become an expectation, if you happen to finish a project on time this way, it'll be used as "See, no problem here, you got it done!", pat on the back and condescending compliments ensue.

So you realize you're never given time to do decent work, that at the end of the day you're better off not meeting deadlines, and that whatever happens, you're always on the losing side.

I'm not saying this is the case in all companies, but I've also seen this far too often. On the very few occasions I had the chance to be the main point of contact with clients, projects were on time, clients were satisfied, and of course, some sales guy still slapped his name somewhere on there and got his bonus, but at least I got to do good work on reasonable deadlines.

I loathe most sales and marketing people, I often find they're a completely unnecessary layer in product and value creation. It's not that their position couldn't be of value, but I don't believe it can be if the money incentive is always more prevalent than the idea of good work done or happy people. As Henry Ford said, "A business that makes nothing but money is a poor business".

Note: I've resigned from this job to work in research.

I have been in a company where we developed hardware and software. Well hardware - you have some hard stuff there - at the end the laws of physics give you a lower bound of stupidity that you can deal with. There are just times when you can write in reply to the manager with a cc to the CEO when answering the question - can we do it that way - with "Only if you send us to training in Hogwarts"

Because software is magic for a lot of people anyway that excuse just doesn't work.

In this analogy, you have a single destination; anything short of that is useless.

What about the agile idea that you deliver something incrementally useful with every sprint?

Obviously, some projects can't be done that way. But if they can, it would mean 1) estimation can be done on a 2-week scale instead of maybe a 6-month one, and 2) they can start deriving benefit/making profit from what you've built while you build the next thing.

More like,

1) When someone buys custom built hardware, they know what they want, and the specifications for it. When someone buys software, they want "something that does xyz, is just like facebook and paypal, but also has our logo." And then it turns out that the problem they're trying to solve is completely unrelated to any of those things. But you can just rewrite it to do those things too, right?

2) If there's a problem with the hardware, the customer that paid the money for custom chips will notice, and will have the technical skill to lay the blame on the vendor accurately. This means hardware manufacturers need to deliver quality, and since point #1 guarantees they actually have a specification, they can therefore engineer to it. In software, often the client takes your application, puts it on machines from the mid 90s that it won't run on, gives it to minimum wage employees to use (so management may not really care if it works), and then ask if its too late for the product to also be an app on their iphone. When they can't do anything with it 3 years later because it was written by people who have never looked at a computer before, the customer will just say "software always has bugs / always going out of date, find a new consulting company to write us a new one", as if its normal to have to build a completely new system to do the same thing over and over again every three years.

Kahnemann and Tversky call it the "planning fallacy".[1] the interesting bit is that the predictions are pessimistic when you get "non-involved" parties to estimate time to completion of the same task. Perhaps using a figure between self estimates and third party estimates may work?

[1] http://en.wikipedia.org/wiki/Planning_fallacy