The forecasting fallacy (2020)

Also to your point, it makes no sense to say, "You cannot predict it. But you can create it." The reason a person or entity creates something is that they've predicted a desired outcome for that creation.

> It seems that we need to figure out what separates the kind of prediction that is the engine of human life and progress from the kind that is just useless blather.

For sure. The author treats content marketing by management consultancies — blather — as serious efforts to predict outcomes. But these are stories about potential outcomes created to lure customers to the rim of their sales funnel. In other words, their actual prediction is that publishing thousands of "thought leadership" pieces will improve SEO and sales engagement, which is probably true.

It's like saying if you drive tomorrow you're going to get in a fatal accident, no one in their right mind would drive in that case.

The only way it can work is if you make the prediction and don't tell those that are affected. But generally in any larger market attempting to capitalize on the future state of the market changes the market and the predicted position.

The people who were working on Y2K did know what would happen in many cases. Their work avoided known huge messes in banking, infrastructure, aviation, and healthcare, among others.

What they didn’t do, much, is write or blog about their work. A lot of fixes were to commercial or government systems running on commercial or government hardware. Publicly disclosing problems and fixes was not part of those cultures.

So it is very hard, today, for members of the public to go back and reconstruct the problems and solutions to “prove” that there were real issues. Which has led some people to believe, incorrectly, that there were not real issues.

Much of the discussion here including the linked article fail to make an important distinction between domains. Prediction can be done quite effectively on thin tailed processes. A lot of the counter examples listed in the comments here are physical systems which are thin tailed. I see aircraft autopilot, collision detection, ozone depletion. These are all well understood physical phenomenon in which large deviations do not occur — your car doesn’t get teleported elsewhere in the middle of avoiding a collision. If a large deviation did occur, say a meteor striking between your car and the object it is attempting to avoid, the collision avoidance system would almost certainly fail. These events occur so infrequently that the system can just assume they won’t and boast a high success rate.

Meanwhile the examples from the linked article are fat tailed processes. Recessions, GDP, interest rates, exchange rates. These are all subject to large discontinuous jumps. Anyone doing a 5 year rate prediction in July 2019 would have been required to predict the pandemic in order to accurately forecast. This is a single example but predictions in this domain are regularly blown out by being teleported to a completely different world. Unlike the thin tailed domain these events happen frequently enough that they’re the only thing that matters for the forecast.

Knowing which class your generating process belongs to is critical to understanding whether forecasting will be effective or not. I’ll take collision detection and leave economic forecasts at the door any day.

Ugh. This annoys me.

Can we predict things super accurately? Often no. But you know what’s better than anecdotes about times predictions were bad? Training and testing sets to judge how good we expect models and predictions to be from the beginning. Because a lot of these are not high confidence predictions.

And no, it’s not “black swans”. Are those a thing? Sure. It’s ok that models can’t account for things that are not modeled or seen before. But if these things are common enough that they’re systemic and the mode is just not actually accommodating for the world of relevant factors, then it’s not going to have been a good model on the test set to begin with. And we would know that.