Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forecasts need to have error bars

andrewpwheeler.com

Post reply on HN
Print view View original

151–160 of 165 posts

Re: Forecasts need to have error bars

#151

Post by 3abiton »

borg16 wrote: post #146

Reminds me of this paper[1] > An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability > Traditionally, scientists have placed more emphasis on communicating inferential uncertainty (i.e., the precision of statistical estimates) compared to outcome variability (i.e., the predictability of individual outcomes). Here, we show that this can lead to sizable…

Very interesting reference, there is a whole field about Uncertainty modelling.

Re: Forecasts need to have error bars

#152

Post by jsmith45 »

kqr wrote: post #76
nullindividual wrote: post #4

Earlier quoted context omitted.

Same, but in a human context, are mundane atmospheric events so far off today that error bars would have any practical value and/or potentially introduce confusion?

Sure -- just a few day outs the forecast is not much better than the climatological average -- see e.g. https://charts.ecmwf.int/products/opencharts_meteogram?base_... Up until that point, error bars increase. At least to me, there's a big difference between "1 mm rain guaranteed" and "90 % chance of no rain but 10 % chance of 10 mm rain" but both have the same average.

The other thing is that many forecasts have a large range. So it might be that the left third of the forecast area has like a 99.9% chance of zero precipitation, the right third has a 99.9% chance of some precipitation, and there is some uncertainly about where the border will be, (and near the border might have less precipitation than the edge).

The result is that I will be told ~50% chance of precipitation, but the places I care about might well be in the essentially 0% or essentially 100% parts.

The single forecast for a decent size city problem impacts other parts like forecasted highs/low. Even without a front, in some cities that have incorporated much of their suburbs, it is not uncommon for the eastmost part and westmost part to differ by 5-6 degrees, and that is ignoring the inherent temperature differences found in the downtown areas.

Re: Forecasts need to have error bars

#153

Post by marcinzm »

staunton wrote: post #108
marcinzm wrote: post #55

Earlier quoted context omitted.

Most decent companies and especially tech do AB testing for everything including having people whose only job is to make sure those test results are statistically valid.

The magic words here are make sure .

In my experience they make sure a ton more in industry than in academia.

Re: Forecasts need to have error bars

#154

Post by gh02t »

kqr wrote: post #147
gh02t wrote: post #142

Earlier quoted context omitted.

Even subjectively, on what basis would they generate uncertainties that at least keeps some grounding in reality? Any human generation would be ad hoc and likely very wrong, humans are notoriously awful at estimating risk and I'd argue by extension uncertainties with any consistency. And that's not even considering how one would assign an uncertainty to some huge model with 350 wacko features trained on 40 million ex…

> on what basis would they generate uncertainties that at least keeps some grounding in reality? By having their forecasts continuously evaluated against outcomes. If someone can show me they have a track record of producing calibrated error bars on a wide variety of forecasts, I trust them to slap error bars on anything. > even if a human is able to slap an uncertainty on a prediction [...] that doesn't mean it's re…

> This sounds like it's approaching some sort of model mysticism. Models don't make forecasts, humans do.

By forecasts I only mean output of a model, I've been wrapped up in time series methods where that's the usual term for model outputs. Assigning confidence to the conclusions drawn by an analyst using some model as a tool is a different task that may or may not roll up formal model output uncertainties and usually involves a lot of subjectivity. This is an important thing too, but is downstream.

Uncertainty is inherently tied to a specific model, since it characterizes how the model propagates uncertainty of inputs and its own fit/structure/assumptions onto its outputs. If you aren't building uncertainties contingent on the characteristics of a specific model then it isn't an uncertainty. But there's no mysticism about models possibly being unintuitive, most of the popular model forms nowadays are mystery black boxes. Some function fit to a specific dataset until it finds a local minimum in a loss function that happens to do a good job (simplifying). There's plenty of work that shows ML models often exploit features and correlations that are highly unintuitive to a human or are just plain spurious.

Re: Forecasts need to have error bars

#155

Post by staunton »

marcinzm wrote: post #153
staunton wrote: post #108

Earlier quoted context omitted.

The magic words here are make sure .

In my experience they make sure a ton more in industry than in academia.

Any anecdotes you can share? Also, I meant "make sure" in a negative way. As in you "make sure things are statistically significant" by e.g. p-hacking. Not that this isn't done in science but I think you're more in danger of being embarrassed during peer review than by the C-suite reading your executive summary...

Re: Forecasts need to have error bars

#156

Post by bo1024 »

cubefox wrote: post #145
bo1024 wrote: post #80

Earlier quoted context omitted.

No, predictive interval is more precise, since we are dealing with predicting an observation rather than forming a belief about a parameter.

What's a predictive interval?

I don't normally use that term, but someone else in reply to me did, and it captures what I wanted to say:

https://en.wikipedia.org/wiki/Prediction_interval

Re: Forecasts need to have error bars

#157

Post by bo1024 »

fjkdlsjflkds wrote: post #122
borroka wrote: post #25

Earlier quoted context omitted.

It is a very common misconception and one of my technical crusades. I keep fighting, but I think I have lost. Not knowing what the "uncertainty interval" represents (is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values?) could be even more dangerous, in theory, than using no uncertainty interval at all. I say in theory because, in my experience in the tech ind…

> Not knowing what the "uncertainty interval" represents (is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values?) could be even more dangerous, in theory, than using no uncertainty interval at all. And, from what I understand, this is what is happening in this article. The person is providing an uncertainty interval for their mean estimator and not for future o…

Thanks, this finally clarifies for me what the article was actually doing!

And it's very different from what I expected, and it doesn't make a lot of sense to me. I guess if statisticians already believe your model, then they want to see the error bars on the model. But I would expect if someone gives me a forecast with "error bars", those would relate to how accurate they think the forecast would be.

Re: Forecasts need to have error bars

#158

Post by bo1024 »

fjkdlsjflkds wrote: post #120
bo1024 wrote: post #81

Earlier quoted context omitted.

Thanks for explaining how it works - I don't use R (I assume this is R). This does not seem like a good way to produce "error bars" around a forecast like the one in this case study. It seems more like a note about how much volatility there has been in the past.

> I don't use R (I assume this is R) Just to clarify... this is Python code, not R.

Thanks.

Re: Forecasts need to have error bars

#159

Post by marcinzm »

staunton wrote: post #155
marcinzm wrote: post #153

Earlier quoted context omitted.

In my experience they make sure a ton more in industry than in academia.

Any anecdotes you can share? Also, I meant "make sure" in a negative way. As in you "make sure things are statistically significant" by e.g. p-hacking. Not that this isn't done in science but I think you're more in danger of being embarrassed during peer review than by the C-suite reading your executive summary...

Most companies that care will run everything though an AB test, the number of AB tests is physically limited by traffic volume and the team in charge of measuring the results of the AB tests is not the team that created the experiment. That makes it much harder to p-hack since you cannot re-run experiments infinitely on a laptop, and the measurement team is judged on the accuracy of their forecasts versus the revenue impact.

Re: Forecasts need to have error bars

#160

Post by gyrovagueGeist »

hgomersall wrote: post #150
8organicbits wrote: post #134

Earlier quoted context omitted.

Model: water freezes below 0° C. Input: temperature is measured at -1° C. Prediction: water will freeze. Actual: water didn't freeze. Actual temperature: 2° C. The model isn't broken, it gives an incorrect result because of input error.

Well I'd say the model is broken because it didn't capture the uncertainty in the measurements.

Taking the example in this comment thread, even if the model takes an arbitrary nonparametric distribution of input temperatures and perfectly returns the posterior distribution of freezing events there is still a difference in model error and forward UQ error.

The model itself can perfectly describe the physics, but it only knows what you can give it. This may be limited by measurement uncertainty of your equipment, etc, but it is separate from the model itself.

In this area, "the model" is typically considered as the input parameter to quantity of interest map itself. It's not the full problem from gathering data to prediction.

Model error would be things like failing to capture the physics (due to approximations, compute limits, etc), intrinsic aleatoric uncertainty in the freezing process itself, etc.

Making this distinction helps talk about where the uncertainty comes from, how it can be mitigated, and how to use higher level models and resampling to understand its impact across the full problem.

Post reply on HN