Why are software development estimates regularly off by a factor of 2-3?

You might end up taking twice as long. You might end up taking half the time estimated. Even with a perfectly honest estimate (which is rare) both scenarios are equally likely. So the overall distribution skews towards late delivery - you are as likely to require 2 more days, as you are to save 1.

Hardware engineering is possible as physics is the boundary and since has become categorically managed by human's mind of structure and logic. A jumbo jet can be decomposed to millions parts each confined by own specification therefore once design is done the production timeline is easy to estimated.

Software production to this day is still struggling between modular development and any attempt to standardize on a way to create software is overwhelmed by thousands of new frameworks or libraries even engineering methodologies every day. The reason software has such "joy" is because not like material or aerodynamic physics, software and way of making software are frequently derailed by its foundation, computer chips new advancement. Literally, every few years whoever can exploit the new capabilities of computer hardware has a chance to become next Steven Jobs.

Granted, most of corporate enterprise software projects don't have to aim high and certainly can be managed like any consumer production manufacture, so long as people are willing to settle on certain software framework, engineering approach, and treat pre-existed software component like real "physical" component meaning quit thinking the possibility of customize anything, simply focus on assembling. This is the engineering propaganda from school of IBM, Microsoft and SAP et al have been trying for decades but reality is no one but the labor cost sensitive offshore software firm would be interested to this. Everyone in corporate world pretend they're working in a creative campus and fancy about the software they make is the core competitive edge of their employer. The price to pay is uncertainty, but the question to ask is if their employer is in the business is in software business. The hard part is even this question is full of uncertainty, who'd imagine Walmart will be compete on the software it runs based on its past business model, well that might change.

The art to be able to estimate accurately (heck, even Boeing failed 787 estimate) is to turn craftsmanship to production assembling line. Stop thinking creatively, focus on manage a collection of proven engineering approaches of "manufacture" features meeting requirement. Everyone in software industry should realize there needs a distinct separation between the creative making of software world and the basic business operation needs. The line is not static though, as Google's engineers constantly made IBM's scalable/reliable enterprise software suite a joke. Still, not everyone is google and effectively manage the evolution of this distinction is not easy but necessary, because while google is looking for uncertainty, most of corporate enterprise hate uncertainty.

Once accepted the confined world, it will be possible to maintain a model of software composition by different combination of modular components, pre-existed or further decompose. The process can be fully automated and emulated as long as the time factor is added to each component. The upside is this can be estimate/managed down to minute level yet the downside is that the software created would be boring lack lust, but built with complexity to support operation reliably which is enterprise software all about anyway. Most importantly, once the estimate and cost become predictable, managing software system evolution will become a much easier and lucrative job. Imaging how joyful will it become if you go to your CFO asking for next round of upgrade (or even revamp) due to the "demand" of some fancy new capability the business users have experienced from their personal consumer digital gadget, while your commitment to the money and time has a solid track record.

The key to resolution to the challenge of corporate software engineering, including the majority of creative product builder in start ups, is the lack of an effective and efficient way to manage the software evolution and the chaos it brought. Sourcing to less cost labor or adding more bodies to the equation will not make it better, except short term psychological benefit.

In sum, software industry desperately needs automation. Ironically, software starts as the automation king to everything else, but itself has been largely remained in the hand of artist. The good news is cloud and DevOp are closing fast to bring an end to the nightmare of software engineering, at least an interim until the day software is no longer made by human.

To many in Management and accounting (and with varying degrees of pointy-haired-ness), profit centers "make money" while cost centers "spend/burn/"waste" money.

The sun shines on the former, while the inquisition perpetually stalks the latter.

Systems and systems software are often seen as a cost center. There is constant pressure to "control" these costs and to do more with less.

People are required to be over-optimistic in their communication while at the same time overly pessimistic ("frugal") in their resource allocation. Note the use of the word "pessimism". It is not so much a matter of "optimisim" thinking that "we can get it done with less"; rather, it is the pessimism of "we can't afford that".

Much of my professional life has been spent in big, "conservative" (cough), risk-averse organizations. I'm sure things are to some degree different elsewhere -- some degree.

Nonetheless, it's worth keeping in mind this (simplistic, and to varying degrees counter-productive) concept of "profit centers" and "cost centers". And if you are working for the latter, expect to be squeezed into ever more unrealistic positions.

It's something the seasoned professional stays attuned to. There will always be some squeezing; when it starts to worsen and become excessive, it's time to go. Your personal solution, as a "simple" employee, is to find another organization -- the sooner, the better.