Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forecasts need to have error bars

andrewpwheeler.com

Post reply on HN
Print view View original

121–130 of 165 posts

Re: Forecasts need to have error bars

#121

Post by hgomersall »

eru wrote: post #119
hgomersall wrote: post #117

Earlier quoted context omitted.

Many economists are so fully bought into their models that they can't think of any alternatives, despite them being essentially useless. I interpreted skin-in-the-game in that way - as professionally committed. Perhaps something different was meant.

> Many economists are so fully bought into their models [...] How do you know that? Whenever I interact with economists, mostly online via blogs but also sometimes via email, they always seem painfully aware of the shortcomings of their models, and don't seem to confuse them with reality. Perhaps you have studied a different sub-population of economists than the ones I have anecdotal experience with?

I'm a sense, that makes my point. Why do they persist with models that don't represent reality despite knowing it? Eventually you must realise that adding epicycles isn't going to cut it, yet still the sage voices echo the standard dogma when economies are dragged into the doldrums by policy posed by useless models.

Bought into is not the same as believing.

Re: Forecasts need to have error bars

#122

Post by fjkdlsjflkds »

borroka wrote: post #25
m-murphy wrote: post #16

Earlier quoted context omitted.

You're probably thinking of a predictive interval

It is a very common misconception and one of my technical crusades. I keep fighting, but I think I have lost. Not knowing what the "uncertainty interval" represents (is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values?) could be even more dangerous, in theory, than using no uncertainty interval at all. I say in theory because, in my experience in the tech ind…

> Not knowing what the "uncertainty interval" represents (is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values?) could be even more dangerous, in theory, than using no uncertainty interval at all.

And, from what I understand, this is what is happening in this article.

The person is providing an uncertainty interval for their mean estimator and not for future observations (i.e., the error bars reflect the uncertainty of the mean estimator, not the uncertainty over observations).

Like you said: before adding error bars, it probably makes sense to think a bit about what type of uncertainty those error bars are supposed to represent.

Re: Forecasts need to have error bars

#123

Post by fjkdlsjflkds »

m-murphy wrote: post #50
borroka wrote: post #25

Earlier quoted context omitted.

It is a very common misconception and one of my technical crusades. I keep fighting, but I think I have lost. Not knowing what the "uncertainty interval" represents (is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values?) could be even more dangerous, in theory, than using no uncertainty interval at all. I say in theory because, in my experience in the tech ind…

Agreed! I also think it's extremely important as practitioners to know what we're even trying to estimate. Expected value (i.e. least squares regression) is the usual first thing to go for, does that even matter? We're probably actually interested in something like an upper quantile for planning purposes. And then the whole model component of it, the interval that's being simultaneously estimated is model driven and…

> We're probably actually interested in something like an upper quantile for planning purposes.

True. But a conditional quantile is much harder to accurately estimate from data than a conditional expectation (particularly if you are talking about extreme quantiles).

Re: Forecasts need to have error bars

#124

Post by eru »

hgomersall wrote: post #121
eru wrote: post #119

Earlier quoted context omitted.

> Many economists are so fully bought into their models [...] How do you know that? Whenever I interact with economists, mostly online via blogs but also sometimes via email, they always seem painfully aware of the shortcomings of their models, and don't seem to confuse them with reality. Perhaps you have studied a different sub-population of economists than the ones I have anecdotal experience with?

I'm a sense, that makes my point. Why do they persist with models that don't represent reality despite knowing it ? Eventually you must realise that adding epicycles isn't going to cut it, yet still the sage voices echo the standard dogma when economies are dragged into the doldrums by policy posed by useless models. Bought into is not the same as believing.

> Why do they persist with models that don't represent reality despite knowing it?

Why do physicists ignore friction whenever possible?

In general, for any task, you take the simplest model that represents the aspects of reality that you care about. But you stay aware of the limits. That's true in physics or engineering just as much as in economics.

That's why NASA uses Newtonian mechanics for all their rocket science needs, even though they have heard of General Relativity.

That's why people keep using models known to have limits.

> [...] sage voices echo the standard dogma [...]

You do know that most of published economics is about the limits of the 'standard dogma'? That's what gets you published. I often wish people would pay more attention to the orthodox basics, but confirming well-known rules isn't interesting enough for the journals.

So if eg you can do some data digging and analysis that can show that maybe under this very specific circumstances restriction on free trade might perhaps increase national wealth, that can get you published. But the observation that most of the time free trade, even if the other guy has tariffs, is the optimal policy, is too boring to get published.

Compare also crap like 'Capital in the Twenty-First Century' that catapults its author to stardom with its comparatively boring refutation by orthodox economists that no one cares about.

> [...] dragged into the doldrums by policy posed by useless models.

Most orthodox economics is pretty unanimous about basic policies: for free trade, against occupational licensing, for free migration, for free movement of capital, for simple taxes without loopholes, against messing with the currency, against corruption, against subsidies, for taxes instead of bans (eg on drugs, or emissions, or guns), against price floors or ceilings or other price controls, etc.

Many doldrums happen when policy ignores or contradicts these basic ideas. Alas, economics 101 is not popular with the electorate almost anywhere.

Re: Forecasts need to have error bars

#125

Post by Izkata »

nonethewiser wrote: post #11
Maxion wrote: post #10

Earlier quoted context omitted.

The depressing part is that many people actually need them removed in order to not be confused.

But aren’t they still confused without the error bars? Or confidently incorrect? And who could blame them, when that’s the information they’re given? It seems like the options are: - no error bars which mislead everyone - error bars which confuse some people and accurately inform others

After a lot of back-and-forth some years ago, we settled on a third option: If the error bars would be too big (for whatever definition of "too big" we used back then), don't show the data and instead show a "not enough data points" message. Otherwise, if we were showing the data, show it without the error bars.

Re: Forecasts need to have error bars

#126

Post by lordgrenville »

jalapenos wrote: post #112

Doesn't work. For instance in a business setting, if I say "it'll be done in 10 days +/- 4 days", they'll immediately say "ok so you're saying it'll be done in 14 days tops then". More effective to sound as unsure as possible, disclaim everything in slippery language, and promise to give updates to your predictions as soon as you realise they've changed (granted this wouldn't work as well for an anonymous reader situ…

Is it wrong of them to interpret you that way? What is the correct interpretation?

Re: Forecasts need to have error bars

#127

Post by hgomersall »

eru wrote: post #124
hgomersall wrote: post #121

Earlier quoted context omitted.

I'm a sense, that makes my point. Why do they persist with models that don't represent reality despite knowing it ? Eventually you must realise that adding epicycles isn't going to cut it, yet still the sage voices echo the standard dogma when economies are dragged into the doldrums by policy posed by useless models. Bought into is not the same as believing.

> Why do they persist with models that don't represent reality despite knowing it? Why do physicists ignore friction whenever possible? In general, for any task, you take the simplest model that represents the aspects of reality that you care about. But you stay aware of the limits. That's true in physics or engineering just as much as in economics. That's why NASA uses Newtonian mechanics for all their rocket scienc…

> Most orthodox economics is pretty unanimous about basic policies: for free trade, against occupational licensing, for free migration, for free movement of capital, for simple taxes without loopholes, against messing with the currency, against corruption, against subsidies, for taxes instead of bans (eg on drugs, or emissions, or guns), against price floors or ceilings or other price controls, etc

It's interesting that if you oblige your models to fit a set of policy positions then they return that set of policy positions and are pretty useless in general. A cynic might say that's by design.

Orthodox macroeconomic modelling is laughably naive and mathematically wrong before even getting to the basic issues of failure to validate. Let's not compare it to disciplines where validation is the entire point.

Your rhetoric clearly shows you don't want to think too critically about this so I'll sign off now.

Re: Forecasts need to have error bars

#129

Post by amai »

esafak wrote: post #21

Uncertainty quantification is a neglected aspect of data science and especially machine learning. Practitioners do not always have the statistical background, and the ML crowd generally has a "predict first and asks questions later" mindset that precludes such niceties. I always demand error bars.

Error bars are important. But most people misinterpret their meaning, see https://errorbars.streamlit.app/
Post reply on HN