Why are software development estimates regularly off by a factor of 2-3?

I'm inclined to agree. Often though a client will want an estimate (to a ridiculous accuracy) before agreeing to the work. Unless you churn out web page templates or something equally repetitive it's hard to have that relevant velocity.

I still come up against client managers that don't "believe" in Agile approaches too.

Why is the problem inherently hard? To see why, let's imagine that the All-Knowing Fairy Godmother of Software Estimation descends from the heavens and lends to us her magic estimating function F that, applied to any software project S, will tell us exactly how much time and money our preferred software team will consume to implement S. Don't worry about how F works, just believe that it does. (Okay, okay. Let's just say that F peers into a parallel universe in which our team has already implemented a software system line-for-line identical to S, and it just observes how much time and money were actually consumed in that universe. Anyway...)

Now that we have the magic F, our problem is easy, right? Nope. The real problem is that we don't know what S is. All we actually know is that our S, whatever it ends up being, must satisfy some set of fuzzy constraints C (commonly called "software requirements").

In truth, there are a countless number of possible values for S. In other words, S is a random variable, a mapping that ascribes a probability to each of the possible software systems that we could build to satisfy C. Therefore, our best estimate, even with the perfect estimating function F, is itself a random variable. In other words, it's not an estimate but a distribution of estimates.

See how fun this is getting?

But, wait, it gets funner. That's because people in the business world don't want a distribution of possibilities. They want a budget, a date on the calendar. So we must squeeze point estimates out of the true distribution. (And, remember, we don't even know what the true distribution is!)

That's where the second reason kicks in. Let's just think about the distribution of possible budgets for our distribution of possible values of S. Because S can range from "the simplest thing that could possibly satisfy C" to "the most insanely complex thing that a frighteningly gifted salesperson for an enterprise consulting firm could get our CIO to throw money at," that distribution is going to be w-i-d-e. From X to 10X wide.

So if we're the folks tasked with coming up with those point estimates, we could plausibly estimate X on the low end, or 10X on the high end, or anything in between. Guess which estimates are going to get us the most push-back from the higher-ups?

Now, we're swell guys and all, and we want to do a good job with our estimates. No question. But, still... We're human. If we think that giving a higher-end estimate is going to make us unpopular with the higher-ups, maybe we'll estimate a little lower. And when we get push-back on that estimate, maybe we'll "take another look at the numbers" to see if we "missed any opportunities."

And that's the old one-two. We start with a hard, fuzzy problem and then add to it the pressure to deliver only feel-good solutions. The result: estimates that are often way low.

I think we have different intuitions here because it sounds like you're more in a service business than specifically writing software. I agree that a lot of any service business is standardizable, because it's mainly about people and their needs; that has a lot of regularity.

But I don't think the software creation part of a service business is standardizable over the long term. During the first wave of "put smallish businesses on the internet" each web site was custom, hand-rolled software. Early on those schedules were unpredictable, but for a while, it became a known, predictable business.

That business has, in the long term, been basically destroyed. People spotted the regularities and developed common code and tools. What was mainly a problem of software development became a (much smaller) problem of installation and configuration. The competitive advantage for those people now lies not in coding ability, but in customer service and in helping people manage the essential complexity of the domain.

I would just as soon not keep customers only interested in the lowest price. They're the ones that typically will be the biggest headache.