Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forecasts need to have error bars

andrewpwheeler.com

Post reply on HN
Print view View original

31–40 of 165 posts

Re: Forecasts need to have error bars

#31

Post by nradov »

figassis wrote: post #23
esafak wrote: post #21

Uncertainty quantification is a neglected aspect of data science and especially machine learning. Practitioners do not always have the statistical background, and the ML crowd generally has a "predict first and asks questions later" mindset that precludes such niceties. I always demand error bars.

So is it really science? These are concepts from stats 101. And the reasons and need, and the risks of not having them are very clear. But you have millions being put into models without these pre-requisites, and being sold to people as solutions, and waved away as "if people buy is it's bc it has value". People also pay fraudsters.

Mostly not. Very few data "scientists" working in industry actually follow the scientific method. Instead they just mess around with various statistical techniques (including AI/ML) until they get a result that management likes.

Re: Forecasts need to have error bars

#32

Post by ta8645 »

Maxion wrote: post #10
amichal wrote: post #5

I have, in my life as a web developer, had multiple "academics" urgently demand that i remove error bands, bars, notes about outliers, confidence intervals etc from graphics at the last minute so people are not "confused" Its depressing

The depressing part is that many people actually need them removed in order to not be confused.

That is baldly justifying a feeling of superiority and authority over others. It's not your job to trick other people "for their own good". Present honest information, as accurately as possible, and let the chips fall where they may. Anything else is a road to disaster.

Re: Forecasts need to have error bars

#33

Post by alistairSH »

nonethewiser wrote: post #11
Maxion wrote: post #10

Earlier quoted context omitted.

The depressing part is that many people actually need them removed in order to not be confused.

But aren’t they still confused without the error bars? Or confidently incorrect? And who could blame them, when that’s the information they’re given? It seems like the options are: - no error bars which mislead everyone - error bars which confuse some people and accurately inform others

Yep.

See also: Complaints about poll results in the last few rounds of elections in the US. "The polls said Hillary would win!!!" (no, they didn't).

It's not just error margins, it's an absence of statistics of any sort in secondary school (for a large number of students).

Re: Forecasts need to have error bars

#34

Post by knicholes »

sonicanatidae wrote: post #28
esafak wrote: post #20

Earlier quoted context omitted.

Statistically illiterate people should not be making decisions. I'd take that as a signal to leave.

Statistically speaking, you're in the minority. ;)

Maybe not in the minority for taking it as a signal to leave, but in the minority for actually acting on that signal.

Re: Forecasts need to have error bars

#35

Post by mightybyte »

Completely agree with this idea. And I would add a corollary...date estimates (i.e. deadlines) should also have error bars. After all, a date is a forecast. If a stakeholder asks for a date, they should also specify what kind of error bars they're looking for. A raw date with no estimate of uncertainty is meaningless. And correspondingly, if an engineer is giving a date to some other stakeholder, they should include some kind of uncertainty estimate with it. There's a huge difference between saying that something will be done by X date with 90% confidence versus three nines confidence.

Re: Forecasts need to have error bars

#36

Post by borroka »

figassis wrote: post #23
esafak wrote: post #21

Uncertainty quantification is a neglected aspect of data science and especially machine learning. Practitioners do not always have the statistical background, and the ML crowd generally has a "predict first and asks questions later" mindset that precludes such niceties. I always demand error bars.

So is it really science? These are concepts from stats 101. And the reasons and need, and the risks of not having them are very clear. But you have millions being put into models without these pre-requisites, and being sold to people as solutions, and waved away as "if people buy is it's bc it has value". People also pay fraudsters.

But even in academia, where supposedly "true science" is, if not done, at least pursued, uncertainty intervals are rarely, with respect to the times they would be needed, understood and used.

When I used to publish stats- and math-heavy papers in the biological sciences, very rarely the reviewers--and I used to publish in intermediate and up journals--were paying any attention to the quality of the predictions, beyond a casual look at the R2 or R2-equivalents and mean absolute errors.

Re: Forecasts need to have error bars

#38

Post by [deleted] »

amichal wrote: post #5

I have, in my life as a web developer, had multiple "academics" urgently demand that i remove error bands, bars, notes about outliers, confidence intervals etc from graphics at the last minute so people are not "confused" Its depressing

[deleted]

Re: Forecasts need to have error bars

#39

Post by cycomanic »

amichal wrote: post #5

I have, in my life as a web developer, had multiple "academics" urgently demand that i remove error bands, bars, notes about outliers, confidence intervals etc from graphics at the last minute so people are not "confused" Its depressing

Can you be more specific (maybe point to a website)? I am trying to imagine the scenarios where a web developer would work with academics and does the data processing for the representation? Of the few scenarios that I could think about where an academic works directly with a web developer they would almost always provide the full figures.

Re: Forecasts need to have error bars

#40

Post by marcosdumay »

nonethewiser wrote: post #11
Maxion wrote: post #10

Earlier quoted context omitted.

The depressing part is that many people actually need them removed in order to not be confused.

But aren’t they still confused without the error bars? Or confidently incorrect? And who could blame them, when that’s the information they’re given? It seems like the options are: - no error bars which mislead everyone - error bars which confuse some people and accurately inform others

Yeah, when people remove that kind of information to not confuse people, they are aiming into making them confidently incorrect.
Post reply on HN