Why are software development estimates regularly off by a factor of 2-3?

I think you're wrong.

Perhaps you weren't around in the early 2000s when the RAD tools were all the rage.

Perhaps you missed out on the colossal frameworks of the late 2000s when everything was a factory and understanding HTTP was actually a disadvantage as the whole thing would blow up if you actually tried to access the request body.

We went down the road you talk of. It was horrible. Now the pendulum swings to the opposite ends, the light-weight APIs which don't try and abstract away all the details which it turns out mattered a lot as everyone has to do pretty much the same thing, but ever-so-slightly differently. Tiny tools that do one job well.

All you are talking about is writing HTML and doing server config, neither of which are software or programming. When someone in 2000 came to you and said 'I need a website, therefore I need a software programmer to write html and setup a server' they no more needed a programmer back than then they needed one today. Just back then it was developers who knew the markup language and how to configure servers and they weren't about to turn away silly money just because there was not much actual coding involved.

Expect to be told, halfway there, that the party has moved to Tijuana. After all you agreed to hike to the party and it would be unreasonable to expect of the host not to move the party when the weather changes.

Reducing the complexity of products is an enticing ideal, but in reality, some products are inherently complex. Let's take one of Steve Jobs' products, iOS, as an example. No matter how much you try to simplify it, it's going to be a huge piece of software (I'm guessing hundreds of developers and supporting staff). Apple's success has depended on keeping customers engaged by regularly coming out with improved versions of products, so they need to estimate fairly accurately what the release dates are going to be: being six months late could be disastrous for marketing efforts and the company's stock price (e.g., you want to get the next version of the iPhone in the stores by Christmas). So you can be sure that there are people at Apple whose job it is to make very careful estimates of how long it will take to develop stuff.

Because no one wants to admit they still have to learn more about the system they are going to build.

It's more like this: you've been on dozens and dozens of hiking trips. Except these trips generally go through seismically unstable territory and war zones. You just never know when you'll have to dig a tunnel underneath a battlefield to get to the next river you'd have to cross — a river unmarked on your map, or perhaps the result of some aggressive "engineering" by someone since your map was drawn.

Then you get an urgent message from your customer: you weren't supposed to hike from SF to LA after all. You actually need to go from New York to London, on a submarine. Specs require that you do this without surfacing or refueling. So you need to design a nuclear reactor, figure out a way to put it on the leaky old U-Boat your customer just happens to have lying around. All within the same timeline as before, and in the same budget.

You've had all this happen to you a few dozen times, so you factor in the time you know you need to build submarines (all just different enough, of course, that you can't reuse parts, even if the customers let you adapt, say, the propeller shafts to a new customer's design: I hear that in software, this is called "DRY" and "using libraries and frameworks" and "refactoring") — and you give an honest estimate: "it'll take eight months of full-time work, and will cost you about X hundred thousand dollars (a bargain for a nuclear submarine, if you ask me)" — and the customer will then go to someone else, leaving you wondering how you'll pay your bills.