Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forecasts need to have error bars

andrewpwheeler.com

Post reply on HN
Print view View original

61–70 of 165 posts

Re: Forecasts need to have error bars

#61

Post by nequo »

hgomersall wrote: post #58
bo1024 wrote: post #56

Earlier quoted context omitted.

To me, this comes back to the question of skin in the game. If you have skin in the game, then you produce the best uncertainty estimates you can (by any means). If you don't, you just sit back and say "well these are the error bars my model came up with".

It's worse than that. Oftentimes the skin in the game provides a motivation to mislead. C.f. most of the economics profession.

This is a pretty sweeping generalization, but if you have concrete examples to offer that support your claim, I’d be curious.

Re: Forecasts need to have error bars

#62

Post by mnky9800n »

bo1024 wrote: post #8

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…

Recently someone on hacker news described statistics as trying to measure how surprised you should be when you are wrong. Big fat error bars would give you the idea that you should expect to be wrong. Skinny ones would highlight that it might be somewhat upsetting to find out you are wrong. I don't think this is an exhaustive description of statistics but I do find it useful when thinking about forecasts.

Re: Forecasts need to have error bars

#63

Post by PeterisP »

bo1024 wrote: post #56
hgomersall wrote: post #42

Earlier quoted context omitted.

Error bars in forecasts can only mean uncertainty your model has. Without error bars over models, you can say nothing about how good your model is. Even with them, your hypermodel may be inadequate.

To me, this comes back to the question of skin in the game. If you have skin in the game, then you produce the best uncertainty estimates you can (by any means). If you don't, you just sit back and say "well these are the error bars my model came up with".

There are ways of scoring forecasts that reward accurate-and-certain forecasts in a manner where it's provably optimal to provide the most accurate estimates for your (un)certainty as you can.

Re: Forecasts need to have error bars

#64

Post by Smoosh »

yakubin wrote: post #27
nullindividual wrote: post #4

Earlier quoted context omitted.

Same, but in a human context, are mundane atmospheric events so far off today that error bars would have any practical value and/or potentially introduce confusion?

Absolutely. 15 years ago I could reasonably trust forecasts regarding whether it’s going to rain in a given location 2 days in advance. Today I can’t trust forecasts about whether it’s raining currently .

It seems unlikely that the modelling and forecasting has become worse, so I guess there is some sort of change happening to the climate making it more unstable and less predictable?

Re: Forecasts need to have error bars

#65

Post by lispisok »

dguest wrote: post #51
doubled112 wrote: post #3

I really thought that this was going to be about the weather.

Me too, and I was looking forward to the thread that talks about error bars in weather models, which is totally a thing! It turns out the ECMWF does do an ensamble model where they run 51 concurrent models, presumably with slightly different initial conditions, or they vary the model parameters within some envelope. From these 51 models you can get a decent confidence interval. But this is a lower resolution model, r…

A lot of weather agencies across the world run ensembles including US, Canada, and the UK. Ensembles are the future of weather forecasting but weather models are so computationally heavy models have a resolution/forecast length tradeoff which is even bigger when trying to run 20-50 ensemble members. You can have a high resolution model that runs to 2 days or so or have a longer range model at much coarser resolution.

ECMWF recently upgraded their ensemble to run at the same resolution as the HRES. The HRES is basically the ensemble control member at this point [1]

[1] https://www.ecmwf.int/en/about/media-centre/news/2023/model-...

Re: Forecasts need to have error bars

#66

Post by kqr »

rented_mule wrote: post #22

Yes, please! I was part of an org that ran thousands of online experiments over the course of several years. Having some sort of error bars when comparing the benefit of a new treatment gave a much better understanding. Some thought it clouded the issue. For example, when a new treatment caused a 1% "improvement", but the confidence interval extended from -10% to 10%, it was clear that the experiment didn't tell us h…

> This makes the decision feel more arbitrary.

This is something I've started noticing more and more with experience: people really hate arbitrary decisions.

People go to surprising lengths to add legitimacy to arbitrary decisions. Sometimes it takes the shape of statistical models that produce noise that is then paraded as signal. Often it comes from pseudo-experts who don't really have the methods and feedback loops to know what they are doing but they have a socially cultivated air of expertise so they can lend decisions legitimacy. (They used to be called witch-doctors, priests or astrologers, now they are management consultants and macroeconomists.)

Me? I prefer to be explicit about what's going on and literally toss a coin. That is not the strategy to get big piles of shiny rocks though.

Re: Forecasts need to have error bars

#67

Post by lispisok »

Smoosh wrote: post #64
yakubin wrote: post #27

Earlier quoted context omitted.

Absolutely. 15 years ago I could reasonably trust forecasts regarding whether it’s going to rain in a given location 2 days in advance. Today I can’t trust forecasts about whether it’s raining currently .

It seems unlikely that the modelling and forecasting has become worse, so I guess there is some sort of change happening to the climate making it more unstable and less predictable?

>I guess there is some sort of change happening to the climate making it more unstable and less predictable?

I've been seeing this question come up a lot lately. The answer is no, weather forecasting continues to improve. The rate is about 1 day improvement every 10 years so a 5 day forecast today is as good as a 4 day forecast 10 years ago.

Re: Forecasts need to have error bars

#68

Post by cubefox »

bo1024 wrote: post #15
m-murphy wrote: post #14

Earlier quoted context omitted.

That's not what a confidence interval is. A confidence interval is a random variable that covers the true value 95% of the time (assuming the model is correctly specified).

Ok, the 'reverse' of a confidence interval then -- I haven't seen a term for the object I described other than misuse of CI in the way I did. ("Double quantile"?)

"Credible interval":

https://en.wikipedia.org/wiki/Credible_interval

Re: Forecasts need to have error bars

#69

Post by kqr »

rented_mule wrote: post #22

Yes, please! I was part of an org that ran thousands of online experiments over the course of several years. Having some sort of error bars when comparing the benefit of a new treatment gave a much better understanding. Some thought it clouded the issue. For example, when a new treatment caused a 1% "improvement", but the confidence interval extended from -10% to 10%, it was clear that the experiment didn't tell us h…

> That caused us to dig a bit deeper and see that the two peaks represented logged-out and logged-in users.

This is extremely common and one of the core ideas of statistical process control[1].

Sometimes you have just the one process generating values that are sort of similarly distributed. That's a nice situation because it lets you use all sorts of statistical tools for planning, inferences, etc.

Then frequently what you have is really two or more interleaved processes masquerading as one. These distributions generate values that within each are sort of similarly distributed, but any analysis you do on the aggregate is going to be confused. Knowing the major components of the pretend-single process you're looking at puts you ahead of your competition -- always.

[1]: https://two-wrongs.com/statistical-process-control-a-practit...

Re: Forecasts need to have error bars

#70

Post by LexGray »

yakubin wrote: post #27
nullindividual wrote: post #4

Earlier quoted context omitted.

Same, but in a human context, are mundane atmospheric events so far off today that error bars would have any practical value and/or potentially introduce confusion?

Absolutely. 15 years ago I could reasonably trust forecasts regarding whether it’s going to rain in a given location 2 days in advance. Today I can’t trust forecasts about whether it’s raining currently .

I think that is a change in definition. 15 years ago it was only rain if you were sure to get drenched. Now rain means 1mm of water hit the ground in your general vicinity. I blame an abundance of data combined people who refuse to get damp and need an umbrella if there is any chance at all.
Post reply on HN