Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forecasts need to have error bars

andrewpwheeler.com

Post reply on HN
Print view View original

51–60 of 165 posts

Re: Forecasts need to have error bars

#51

Post by dguest »

doubled112 wrote: post #3

I really thought that this was going to be about the weather.

Me too, and I was looking forward to the thread that talks about error bars in weather models, which is totally a thing!

It turns out the ECMWF does do an ensamble model where they run 51 concurrent models, presumably with slightly different initial conditions, or they vary the model parameters within some envelope. From these 51 models you can get a decent confidence interval.

But this is a lower resolution model, run less frequently. I assume they don't do this with their "HRES" model (which has twice the spacial resolution) in an ensemble because, well, it's really expensive.

[1]: https://en.wikipedia.org/wiki/Integrated_Forecast_System#Var...

Re: Forecasts need to have error bars

#52

Post by bo1024 »

m-murphy wrote: post #16
bo1024 wrote: post #15

Earlier quoted context omitted.

Ok, the 'reverse' of a confidence interval then -- I haven't seen a term for the object I described other than misuse of CI in the way I did. ("Double quantile"?)

You're probably thinking of a predictive interval

Yes, that term captures what I'm talking about.

Re: Forecasts need to have error bars

#53

Post by bo1024 »

ramblenode wrote: post #30
bo1024 wrote: post #8

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…

> Another is a standard deviation (i.e. you are pretty much predicting the squared difference between your own point forecast and the outcome). What you probably want is the standard error, because you are not interested in how much your data differ from each other but in how much your data differ from the true population.

I don't see how standard error applies here. You are only going to get one data point, e.g. "violent crime rate in 2023". What I mean is a prediction, not only of what you think the number is, but also of how wrong you think your prediction will be.

Re: Forecasts need to have error bars

#54

Post by bo1024 »

borroka wrote: post #25
m-murphy wrote: post #16

Earlier quoted context omitted.

You're probably thinking of a predictive interval

It is a very common misconception and one of my technical crusades. I keep fighting, but I think I have lost. Not knowing what the "uncertainty interval" represents (is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values?) could be even more dangerous, in theory, than using no uncertainty interval at all. I say in theory because, in my experience in the tech ind…

> is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values

If you don't mind typing it out, what do you mean formally here?

Re: Forecasts need to have error bars

#55

Post by marcinzm »

nradov wrote: post #31
figassis wrote: post #23

Earlier quoted context omitted.

So is it really science? These are concepts from stats 101. And the reasons and need, and the risks of not having them are very clear. But you have millions being put into models without these pre-requisites, and being sold to people as solutions, and waved away as "if people buy is it's bc it has value". People also pay fraudsters.

Mostly not. Very few data "scientists" working in industry actually follow the scientific method. Instead they just mess around with various statistical techniques (including AI/ML) until they get a result that management likes.

Most decent companies and especially tech do AB testing for everything including having people whose only job is to make sure those test results are statistically valid.

Re: Forecasts need to have error bars

#56

Post by bo1024 »

hgomersall wrote: post #42
bo1024 wrote: post #8

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…

Error bars in forecasts can only mean uncertainty your model has. Without error bars over models, you can say nothing about how good your model is. Even with them, your hypermodel may be inadequate.

To me, this comes back to the question of skin in the game. If you have skin in the game, then you produce the best uncertainty estimates you can (by any means). If you don't, you just sit back and say "well these are the error bars my model came up with".

Re: Forecasts need to have error bars

#57

Post by pacbard »

bo1024 wrote: post #8

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…

As far as error bars are concerned, you could report some% credible intervals calculated from taking the some%tile out of your results. It’s somewhat Bayesian thinking but it will work better than confidence intervals.

The intuition would be that some% of your forecasts are between the bounds of the credible interval.

Re: Forecasts need to have error bars

#58

Post by hgomersall »

bo1024 wrote: post #56
hgomersall wrote: post #42

Earlier quoted context omitted.

Error bars in forecasts can only mean uncertainty your model has. Without error bars over models, you can say nothing about how good your model is. Even with them, your hypermodel may be inadequate.

To me, this comes back to the question of skin in the game. If you have skin in the game, then you produce the best uncertainty estimates you can (by any means). If you don't, you just sit back and say "well these are the error bars my model came up with".

It's worse than that. Oftentimes the skin in the game provides a motivation to mislead. C.f. most of the economics profession.

Re: Forecasts need to have error bars

#59

Post by nonameiguess »

bo1024 wrote: post #53
ramblenode wrote: post #30

Earlier quoted context omitted.

> Another is a standard deviation (i.e. you are pretty much predicting the squared difference between your own point forecast and the outcome). What you probably want is the standard error, because you are not interested in how much your data differ from each other but in how much your data differ from the true population.

I don't see how standard error applies here. You are only going to get one data point, e.g. "violent crime rate in 2023". What I mean is a prediction, not only of what you think the number is, but also of how wrong you think your prediction will be.

Standard error is exactly what the statsmodels ARIMA.PredictionResults object actually gives you and the confidence interval in this chart is constructed from a formula that uses the standard error.

ARIMA is based on a few assumptions. One, there exists some "true" mean value for the parameter you're trying to estimate, in this case violent crime rate. Two, the value you measure in any given period will be this true mean plus some random error term. Three, the value you measure in successive periods will regress back toward the mean. The "true mean" and error terms are both random variables, not a single value but a distribution of values, and when you add them up to get the predicted measurement for future periods, that is also a random variable with a distribution of values, and it has a standard error and confidence intervals and these are exactly what the article is saying should be included in any graphical report of the model output.

This is a characteristic of the model. What you're asking for, "how wrong do you think the model is," is a reasonable thing to ask for, but different and much harder to quantify.

Re: Forecasts need to have error bars

#60

Post by lnwlebjel »

CalChris wrote: post #48

I'm reminded of Walter Lewin's analogous point about measurements from his 8.01 lectures: any measurement that you make without any knowledge of the uncertainty is meaningless https://youtu.be/6htJHmPq0Os You could say that forecasts are measurements you make about the future.

To that point, similarly:

"Being able to quantify uncertainty, and incorporate it into models, is what makes science quantitative, rather than qualitative. " - Lawrence M. Krauss

From https://www.edge.org/response-detail/10459

Post reply on HN