The forecasting fallacy (2020) • Hacker News Board

Re: The forecasting fallacy (2020)

#11

Post by kbrkbr » Sun, Dec 31, 2023, 2:22 PM UTC

btilly wrote: post #7

kbrkbr wrote: post #6

I think this article has two shortcomings that make its sweeping conclusions shaky. First, it identifies forecasting with point forecasting. There are other ways to put forecasting questions, e.g. lower and upper level with a certain probability. Also it mentions Tetlock, but only his negative findings, not his positive ones that lead to Good Judgement Project, which suggest the contrary of the conclusion of this art…

Tetlock paints a different picture of his positive findings than you do. Specifically, Tetlock's project opens with key issues of scope about what to even try to forecast. Based on his previous work in expert prediction, he concluded that geopolitics is sufficiently chaotic to be impossible to predict 10 years out. So while he did a lot of work on forecasting, it is generally focused on the next year or to. Which mea…

I agree with the assessment that there are not many systems we can predict 10 years out with great confidence, specifically geopolitics.

But I do not think I painted much of a picture of Tetlock's results.

I read the article as concluding: let's stop predicting, it does not work. Let's start building. (After stating we cannot predict this, and we cannot predict that.)

And I think Tetlock"s result contradict that, as I said. Sometimes and under certain circumstances we can predict quite well.

Re: The forecasting fallacy (2020)

#12

Post by INGELRII » Sun, Dec 31, 2023, 2:36 PM UTC

A forecasting system in aircraft autopilot that can accurately forecast when the plane hits the mountain is always wrong.

Forecasting when the forecast depends on the actions of agents that can be informed by the forecast changes the game. If the Fed model forecasts recession and the Fed takes action to prevent it from happening, it changes everything. Only a forecasting model that is not observed/believed by policy makers can predict without intervention.

Layman's idea of forecasting: Predict what happens in the future.

Economic forecasting: Forecast is input for actions. Predict what happens in the future, using this model, these variables, and everything else stay the same. You can check afterward if the model is an accurate forecaster by removing the changes caused by variables outside the model.

Re: The forecasting fallacy (2020)

#13

Post by qznc » Sun, Dec 31, 2023, 2:41 PM UTC

sega_sai wrote: post #9

Any forecasts that don't involve probabilities or confidence intervals are useless. Also, if any forecasters were really serious, they would register their past forecasts and show how good they have been in the past. But I think most of forecasters are probably afraid of showing their true track record.

I agree. The article succumbs to a False Binary Fallacy, where forecasts are either correct or wrong. The real question about forecasts is how certain they are.

https://en.wikipedia.org/wiki/False_dilemma

Re: The forecasting fallacy (2020)

#14

Post by jxf » Sun, Dec 31, 2023, 2:51 PM UTC

The author would also conclude:

* Collision avoidance systems are terrible at forecasting collisions because they almost never result in a collision. (The point of the system is to help you avoid an upcoming collision.)

* The prediction that Y2K would happen was a bad one since it didn't happen. (We spent billions of dollars to make sure it didn't.)

* The 1978 prediction that the ozone layer would be depleted by 2010 was a bad one since it didn't happen. (Humans took action to reverse CFCs and the ozone layer began to regenerate.)

When you make a forecast about an event wherein agents can change the course of the event, the correct evaluation of the forecast is not "did the event happen?" but "would the event have happened but for intervention?".

The author seems to miss this larger point.

Re: The forecasting fallacy (2020)

#15

Post by notahacker » Sun, Dec 31, 2023, 3:08 PM UTC

Zolde wrote: post #10

One should ask economists what a recession is, not how to predict one. Good modelers do not necessarily need (or want) to know what they are predicting and still beat "domain experts". Authority without clear track-record is a net negative to getting good results. It is better to stick to anonymity, and only let the track-record do the talking/weighting. Without a clear track-record it does not even matter if the pre…

> One should ask economists what a recession is, not how to predict one.

Most economists would agree. It's everyone else that says "well if you know so much about how shocks and policy changes cause recessions, why can't you tell me if there will be a recession in $country in Q2 2025?". And in economics, "skin in the game" means policy responses to avoid dire forecast outcomes (or lack of them when nobody expect oil prices to change or a major bank to collapse).

There's no shortage of opportunity to make money by beating everyone else at the prediction game, but the funds that have consistently profited from spotting the recessions ahead of everyone else don't exist any more than the always-right public expert forecasters.

Re: The forecasting fallacy (2020)

#16

Post by cesaref » Sun, Dec 31, 2023, 3:13 PM UTC

If you want a counter example, go and investigate algo trading hedge funds - you'll find they do a pretty solid job of predicting the future. Sure, some of them predict only a few ms into the future, some a few minutes (the one I worked for was in that category) and others will do interday strategies.

I'm pretty sure there are examples which have a track record of decent returns above the markets they trade in with longer term strategies.

So, i'd say there are examples of forecasting working, but generally the people who are good at it don't write about it, and instead use their knowledge and ideas to quietly make money from their insights :)

Re: The forecasting fallacy (2020)

#17

Post by hef19898 » Sun, Dec 31, 2023, 3:18 PM UTC

Zolde wrote: post #10

One should ask economists what a recession is, not how to predict one. Good modelers do not necessarily need (or want) to know what they are predicting and still beat "domain experts". Authority without clear track-record is a net negative to getting good results. It is better to stick to anonymity, and only let the track-record do the talking/weighting. Without a clear track-record it does not even matter if the pre…

Consultancies predicting something isn't forecasting, it is marketing.

And there or only a rare few thing I disagree more stongly with the statement, that good modellers / data scientist / whatever only need knowledge about how to model stuff to beat domain experts. It takes domain experts to judge whether or not a model correct, to identify the known and unknown unknowns and limitations of these models. Claiming otherwise is deeply arrogant, and it ended in disaster everytime I saw it tried. Good modellers need enough domain knowledge to properly work with, and understand, domain experts. And domain experts need sufficient knowledge about modelling to do the same. Both need the willingness to do so. And every modeller needs to accept that reality beats models, always.

Re: The forecasting fallacy (2020)

#18

Post by hef19898 » Sun, Dec 31, 2023, 3:20 PM UTC

qznc wrote: post #13

sega_sai wrote: post #9

Any forecasts that don't involve probabilities or confidence intervals are useless. Also, if any forecasters were really serious, they would register their past forecasts and show how good they have been in the past. But I think most of forecasters are probably afraid of showing their true track record.

I agree. The article succumbs to a False Binary Fallacy, where forecasts are either correct or wrong. The real question about forecasts is how certain they are. https://en.wikipedia.org/wiki/False_dilemma

First rule of forecasts in supply chain management: the forecast is always wrong. And still people ignore that cardinal, and a lotnof smaller, rules all the time.

Re: The forecasting fallacy (2020)

#19

Post by chadash » Sun, Dec 31, 2023, 3:22 PM UTC

cesaref wrote: post #16

If you want a counter example, go and investigate algo trading hedge funds - you'll find they do a pretty solid job of predicting the future. Sure, some of them predict only a few ms into the future, some a few minutes (the one I worked for was in that category) and others will do interday strategies. I'm pretty sure there are examples which have a track record of decent returns above the markets they trade in with l…

I don’t have a background in this but I was under the impression that much of algorithmic trading is that there are trillions of pennies lying around and if you have an algorithm that picks up those pennies faster than anyone else, you make a lot of money. So it’s capitalizing on tiny market inefficiencies rather than directional predictions.

Re: The forecasting fallacy (2020)

#20

Post by Zolde » Sun, Dec 31, 2023, 3:23 PM UTC

INGELRII wrote: post #12

A forecasting system in aircraft autopilot that can accurately forecast when the plane hits the mountain is always wrong. Forecasting when the forecast depends on the actions of agents that can be informed by the forecast changes the game. If the Fed model forecasts recession and the Fed takes action to prevent it from happening, it changes everything. Only a forecasting model that is not observed/believed by policy…

It is always harder to accurately forecast actual recession, than it is to forecast the predictions of the Fed model. You don't need an information edge there, just information parity.

When the Fed takes action, it is usually a very rational action, with a clear-defined goal of long-term economic health. This makes their actions easier to predict than other market participants.

So you went the hard route, forecasting the highly complex system directly, but then "variables outside the model" caused the "accurate" model to not perform well? You don't buy anything with that, since you live in a world with outside variables which mess up your predictions. The solution is to make your model actually accurate, by incorporating these "variables outside the model": Predict what others will predict.