Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forecasts need to have error bars

andrewpwheeler.com

Post reply on HN
Print view View original

91–100 of 165 posts

Re: Forecasts need to have error bars

#91

Post by bo1024 »

nonameiguess wrote: post #59
bo1024 wrote: post #53

Earlier quoted context omitted.

I don't see how standard error applies here. You are only going to get one data point, e.g. "violent crime rate in 2023". What I mean is a prediction, not only of what you think the number is, but also of how wrong you think your prediction will be.

Standard error is exactly what the statsmodels ARIMA.PredictionResults object actually gives you and the confidence interval in this chart is constructed from a formula that uses the standard error. ARIMA is based on a few assumptions. One, there exists some "true" mean value for the parameter you're trying to estimate, in this case violent crime rate. Two, the value you measure in any given period will be this true…

Another import point of discussion:

> What you're asking for, "how wrong do you think the model is," is a reasonable thing to ask for, but different and much harder to quantify.

This definitely seems to me to be what the original author is motivating: forecasts should have "error bars" in the sense that they should depict how wrong they might be. In other words, when the author writes:

> Point forecasts will always be wrong – a more reasonable approach is to provide the prediction intervals for the forecasts. Showing error intervals around the forecasts will show how Richard interpreting minor trends is likely to be misleading.

The second sentence does not sound like a good solution to the problem in the first sentence.

Re: Forecasts need to have error bars

#92

Post by lr1970 »

bo1024 wrote: post #8

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…

Error bars (either confidence interval or standard deviations) are of little use because they do not tell you how the probability is distributed within the confidence interval band. The holy grail of forecasting is the Probabilistic Forecast that predicts the entire posterior distribution so that you can sample from it generating scenarios or realizations of the underlying random process.

Re: Forecasts need to have error bars

#93

Post by americast »

lr1970 wrote: post #92
bo1024 wrote: post #8

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…

Error bars (either confidence interval or standard deviations) are of little use because they do not tell you how the probability is distributed within the confidence interval band. The holy grail of forecasting is the Probabilistic Forecast that predicts the entire posterior distribution so that you can sample from it generating scenarios or realizations of the underlying random process.

While I agree, we can always have multiple overlapping error bars to understand how the probability is distributed. I am not sure how a probabilistic forecast method is able to perform this better because the confidence interval is always generated through sampling in either situation.

Though probabilistic forecasting methods may have a Bayesian approach, it is Monte Carlo sampling that helps generate the confidence intervals.

Feel free to correct me if I am wrong! Thanks :)

Re: Forecasts need to have error bars

#95

Post by aidenn0 »

datadrivenangel wrote: post #2

The interesting example in this article is nowcasting! The art of forecasting the present or past while you're waiting for data to come in. It's sloppy science / statistics to not haven error ranges.

Makes sense that such a thing would exist, though it renders less funny my joke: "I'm almost clairvoyant; I can predict things shortly after they happen"

Re: Forecasts need to have error bars

#96

Post by sobriquet9 »

gh02t wrote: post #72
esafak wrote: post #21

Uncertainty quantification is a neglected aspect of data science and especially machine learning. Practitioners do not always have the statistical background, and the ML crowd generally has a "predict first and asks questions later" mindset that precludes such niceties. I always demand error bars.

You can demand error bars but they aren't always possible or meaningful. You can more or less "fudge" some sort of normally distributed IID error estimate onto any method, but that doesn't necessarily mean anything. Generating error bars (or generally error distributions) that actually describe the common sense idea of uncertainty can be quite theoretically and computationally demanding for a general nonlinear model…

Conformal prediction solves that problem. Split conformal and Jackknife+ are two simplest examples.

Re: Forecasts need to have error bars

#99

Post by ISL »

bo1024 wrote: post #8

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…

A position espoused by Bill Phillips [1], and to which I now adhere:

"You should be willing to take either side of the bet that confidence interval implies." (paraphrasing; he says it better).

For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval.

Aside from being generally correct, this approach is immediately actionable by making the meaning more visceral in discussions of uncertainty. Done right, it pushes you to assign uncertainties that are neither too conservative nor too optimistic.

If the notion of letting your reader take either side of the bet makes your stomach a little queasy, you're on the right track. The feeling will subside when you're pretty sure you got the errorbar right and your reasoning is documented and defensible.

Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course)

[1] https://www.nobelprize.org/prizes/physics/1997/phillips/fact...

Re: Forecasts need to have error bars

#100

Post by eru »

ISL wrote: post #99
bo1024 wrote: post #8

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…

A position espoused by Bill Phillips [1], and to which I now adhere: "You should be willing to take either side of the bet that confidence interval implies." (paraphrasing; he says it better). For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval. Aside fro…

> Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course)

Also assuming normal distribution, I think?

> If the notion of letting your reader take either side of the bet makes your stomach a little queasy, you're on the right track. The feeling will subside when you're pretty sure you got the errorbar right and your reasoning is documented and defensible.

> For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval.

I would like to build some edge into my bets. If a reader takes both sides of your example, they would be come out exactly even.

But since readers are not forced to take any side at all, they will only take the bet if one of the sides has an advantage.

So I would like to be able to say, 20:1 payout the true value is inside the error bar, and 1:10 payout it's outside the error bar (or something like that).

The tighter the spread I am willing to quote, the more confident I am that I got the error estimates right. (I'm not sure how you translate these spreads back into the language of statistics.)

Post reply on HN