Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forecasts need to have error bars

andrewpwheeler.com

Post reply on HN
Print view View original

111–120 of 165 posts

Re: Forecasts need to have error bars

#111

Post by yakubin »

LexGray wrote: post #70
yakubin wrote: post #27

Earlier quoted context omitted.

Absolutely. 15 years ago I could reasonably trust forecasts regarding whether it’s going to rain in a given location 2 days in advance. Today I can’t trust forecasts about whether it’s raining currently .

I think that is a change in definition. 15 years ago it was only rain if you were sure to get drenched. Now rain means 1mm of water hit the ground in your general vicinity. I blame an abundance of data combined people who refuse to get damp and need an umbrella if there is any chance at all.

No. I mean getting drenched, when forecast predicted no rain. 1mm isn’t really noticeable by humans.

Re: Forecasts need to have error bars

#112

Post by jalapenos »

Doesn't work.

For instance in a business setting, if I say "it'll be done in 10 days +/- 4 days", they'll immediately say "ok so you're saying it'll be done in 14 days tops then".

More effective to sound as unsure as possible, disclaim everything in slippery language, and promise to give updates to your predictions as soon as you realise they've changed (granted this wouldn't work as well for an anonymous reader situation like in this article).

Re: Forecasts need to have error bars

#113

Post by earthscienceman »

lispisok wrote: post #65
dguest wrote: post #51

Earlier quoted context omitted.

Me too, and I was looking forward to the thread that talks about error bars in weather models, which is totally a thing! It turns out the ECMWF does do an ensamble model where they run 51 concurrent models, presumably with slightly different initial conditions, or they vary the model parameters within some envelope. From these 51 models you can get a decent confidence interval. But this is a lower resolution model, r…

A lot of weather agencies across the world run ensembles including US, Canada, and the UK. Ensembles are the future of weather forecasting but weather models are so computationally heavy models have a resolution/forecast length tradeoff which is even bigger when trying to run 20-50 ensemble members. You can have a high resolution model that runs to 2 days or so or have a longer range model at much coarser resolution.…

I'm someone working in the field with weather modelers/forecasters but not a modeler myself. Reading this discussion has been an absolute treat ...

The most fascinating thing about the concept of error in models, including in ensembles, is you can only calculate and propagate error for contributors that you can quantify. There are many unquantifiable sources of error. Imagine a physical process that you are unaware of that propagates as a bias, for example ice nucleation via aerosols. Perhaps you don't even model aerosols. How do you account for error here? What does error even mean?

Ensembles only show you intramodel variability. Which is like error, sort of, but only really represents a combination of "real" variability in initial conditions and how that propagates through your physics/parameterizations.

"models" the HN commentators make for their businesses surely have parallel concepts, but I don't see anyone talking about them. Only discussion about the errors you know when the ugliest errors are the ones that no one knows.

Re: Forecasts need to have error bars

#114

Post by eru »

jrumbut wrote: post #106
eru wrote: post #100

Earlier quoted context omitted.

> Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course) Also assuming normal distribution, I think? > If the notion of letting your reader take either side of the bet makes your stomach a little queasy, you're on the right track. The feeling will subside when you're pretty sure yo…

> Also assuming normal distribution, I think? 95% is 95% regardless of the distribution. > I would like to build some edge into my bets. If a reader takes both sides of your example, they would be come out exactly even. You can imagine yourself being equally unhappy to take either side of the bet, if that's easier than imagining yourself being happy to take either side. It is for me, which is probably something to br…

> 95% is 95% regardless of the distribution.

Standard deviation away from the mean don't correspond to the same percentiles for all distributions, or do they?

If you want to be (almost) independent of distribution, you need Chebyshev's inequality. But that one is far weaker.

> Its practical usage is similar to the 68–95–99.7 rule, which applies only to normal distributions. Chebyshev's inequality is more general, stating that a minimum of just 75% of values must lie within two standard deviations of the mean and 88.89% within three standard deviations for a broad range of different probability distributions.[1][2]

https://en.wikipedia.org/wiki/Chebyshev%27s_inequality

> I also think that framing things as bets brings in all the cultural baggage around gambling and so it isn't always helpful. I'm not sure what a better framing is though.

Underwriting insurance without going bankrupt, perhaps?

Re: Forecasts need to have error bars

#115

Post by dguest »

lispisok wrote: post #65
dguest wrote: post #51

Earlier quoted context omitted.

Me too, and I was looking forward to the thread that talks about error bars in weather models, which is totally a thing! It turns out the ECMWF does do an ensamble model where they run 51 concurrent models, presumably with slightly different initial conditions, or they vary the model parameters within some envelope. From these 51 models you can get a decent confidence interval. But this is a lower resolution model, r…

A lot of weather agencies across the world run ensembles including US, Canada, and the UK. Ensembles are the future of weather forecasting but weather models are so computationally heavy models have a resolution/forecast length tradeoff which is even bigger when trying to run 20-50 ensemble members. You can have a high resolution model that runs to 2 days or so or have a longer range model at much coarser resolution.…

Even with all the hype around deep learning and GPUs, the weather services are building a lot of new, CPU-only supercomputers [1]. There are plans to use more GPUs and ML in the coming years, but the weather models are real, tangible, economically impactful and strategically essential products. It's pretty inspiring to see governments dumping a huge amount of computing power into the working models, and also funding research ideas.

[1]: https://www.mdpi.com/2073-431X/11/7/114

Re: Forecasts need to have error bars

#117

Post by hgomersall »

eru wrote: post #102
hgomersall wrote: post #58

Earlier quoted context omitted.

It's worse than that. Oftentimes the skin in the game provides a motivation to mislead. C.f. most of the economics profession.

How do economists have skin in the game? Many of them eg work in universities and some even have tenure. There's not much skin in the game between any forecasts they might make and their academic prospects. Economists working for companies often have to help them understand micro and macro-economics. Eg (some of) Google's economists help them design the ad auctions. It's relatively easy to figure out for Google how w…

Many economists are so fully bought into their models that they can't think of any alternatives, despite them being essentially useless. I interpreted skin-in-the-game in that way - as professionally committed. Perhaps something different was meant.

Re: Forecasts need to have error bars

#118

Post by hgomersall »

gyrovagueGeist wrote: post #109
hgomersall wrote: post #42

Earlier quoted context omitted.

Error bars in forecasts can only mean uncertainty your model has. Without error bars over models, you can say nothing about how good your model is. Even with them, your hypermodel may be inadequate.

They can also mean pushed forward uncertainty from input parameters which isn't exactly the same as model error

I'm not sure I see the distinction. Would you mind clarifying?

Re: Forecasts need to have error bars

#119

Post by eru »

hgomersall wrote: post #117
eru wrote: post #102

Earlier quoted context omitted.

How do economists have skin in the game? Many of them eg work in universities and some even have tenure. There's not much skin in the game between any forecasts they might make and their academic prospects. Economists working for companies often have to help them understand micro and macro-economics. Eg (some of) Google's economists help them design the ad auctions. It's relatively easy to figure out for Google how w…

Many economists are so fully bought into their models that they can't think of any alternatives, despite them being essentially useless. I interpreted skin-in-the-game in that way - as professionally committed. Perhaps something different was meant.

> Many economists are so fully bought into their models [...]

How do you know that? Whenever I interact with economists, mostly online via blogs but also sometimes via email, they always seem painfully aware of the shortcomings of their models, and don't seem to confuse them with reality.

Perhaps you have studied a different sub-population of economists than the ones I have anecdotal experience with?

Re: Forecasts need to have error bars

#120

Post by fjkdlsjflkds »

bo1024 wrote: post #81
nonameiguess wrote: post #59

Earlier quoted context omitted.

Standard error is exactly what the statsmodels ARIMA.PredictionResults object actually gives you and the confidence interval in this chart is constructed from a formula that uses the standard error. ARIMA is based on a few assumptions. One, there exists some "true" mean value for the parameter you're trying to estimate, in this case violent crime rate. Two, the value you measure in any given period will be this true…

Thanks for explaining how it works - I don't use R (I assume this is R). This does not seem like a good way to produce "error bars" around a forecast like the one in this case study. It seems more like a note about how much volatility there has been in the past.

> I don't use R (I assume this is R)

Just to clarify... this is Python code, not R.

Post reply on HN