71–80 of 165 posts
I really thought that this was going to be about the weather.
https://content.meteoblue.com/en/research-education/specific...
Uncertainty quantification is a neglected aspect of data science and especially machine learning. Practitioners do not always have the statistical background, and the ML crowd generally has a "predict first and asks questions later" mindset that precludes such niceties. I always demand error bars.
Another, separate, issue that is often neglected is the idea of calibrated model outputs, but that's its own rabbit hole.
Completely agree with this idea. And I would add a corollary...date estimates (i.e. deadlines) should also have error bars. After all, a date is a forecast. If a stakeholder asks for a date, they should also specify what kind of error bars they're looking for. A raw date with no estimate of uncertainty is meaningless. And correspondingly, if an engineer is giving a date to some other stakeholder, they should include…
A date estimation with no error bars cannot be proven wrong. But! If you say "there's a 50 % chance it's done before this date" then you can look back at your 20 most recent such estimations and around 10 of them better have been on time. Otherwise your estimations are not calibrated. But at least then you know, right? Which you wouldn't without the error bars.
Uncertainty quantification is a neglected aspect of data science and especially machine learning. Practitioners do not always have the statistical background, and the ML crowd generally has a "predict first and asks questions later" mindset that precludes such niceties. I always demand error bars.
You can demand error bars but they aren't always possible or meaningful. You can more or less "fudge" some sort of normally distributed IID error estimate onto any method, but that doesn't necessarily mean anything. Generating error bars (or generally error distributions) that actually describe the common sense idea of uncertainty can be quite theoretically and computationally demanding for a general nonlinear model…
Sure, you'll ideally want a calibrated estimator/superforecaster to do it, but they exist and they aren't that rare. Any decently sized organisation is bound to have at least one. They just need to care about finding them.
I really thought that this was going to be about the weather.
I've been using meteoblue for a while now and they tell you how sure they are of their predictions. Right now I can see that they rate their predictability as medium for tomorrow, but high for the day after. https://content.meteoblue.com/en/research-education/specific...
They also have one for precipitation type distribution: https://charts.ecmwf.int/products/opencharts_ptype_meteogram...
I really thought that this was going to be about the weather.
Same, but in a human context, are mundane atmospheric events so far off today that error bars would have any practical value and/or potentially introduce confusion?
Up until that point, error bars increase. At least to me, there's a big difference between "1 mm rain guaranteed" and "90 % chance of no rain but 10 % chance of 10 mm rain" but both have the same average.
"Point forecasts will always be wrong" - true that for continuous data but if you can predict that some stock will go to 2.01x it's value instead of 2x that's still helpful.
https://en.wikipedia.org/wiki/Gaussian_process#Gaussian_proc...
Earlier quoted context omitted.
To me, this comes back to the question of skin in the game. If you have skin in the game, then you produce the best uncertainty estimates you can (by any means). If you don't, you just sit back and say "well these are the error bars my model came up with".
There are ways of scoring forecasts that reward accurate-and-certain forecasts in a manner where it's provably optimal to provide the most accurate estimates for your (un)certainty as you can.
Earlier quoted context omitted.
Ok, the 'reverse' of a confidence interval then -- I haven't seen a term for the object I described other than misuse of CI in the way I did. ("Double quantile"?)
"Credible interval": https://en.wikipedia.org/wiki/Credible_interval