Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forecasts need to have error bars

andrewpwheeler.com

Post reply on HN
Print view View original

81–90 of 165 posts

Re: Forecasts need to have error bars

#81

Post by bo1024 »

nonameiguess wrote: post #59
bo1024 wrote: post #53

Earlier quoted context omitted.

I don't see how standard error applies here. You are only going to get one data point, e.g. "violent crime rate in 2023". What I mean is a prediction, not only of what you think the number is, but also of how wrong you think your prediction will be.

Standard error is exactly what the statsmodels ARIMA.PredictionResults object actually gives you and the confidence interval in this chart is constructed from a formula that uses the standard error. ARIMA is based on a few assumptions. One, there exists some "true" mean value for the parameter you're trying to estimate, in this case violent crime rate. Two, the value you measure in any given period will be this true…

Thanks for explaining how it works - I don't use R (I assume this is R). This does not seem like a good way to produce "error bars" around a forecast like the one in this case study. It seems more like a note about how much volatility there has been in the past.

Re: Forecasts need to have error bars

#82

Post by netsharc »

Animats wrote: post #37

Looking at the graph, changes in this decade are noise. But what happened back in 1990?

Probably no simple answer, but here's a long paper I just found: https://pubs.aeaweb.org/doi/pdf/10.1257/089533004773563485

Another famous hypothesis is the phasing out of lead fuel: https://en.wikipedia.org/wiki/Lead%E2%80%93crime_hypothesis

Re: Forecasts need to have error bars

#83

Post by m-murphy »

bo1024 wrote: post #54
borroka wrote: post #25

Earlier quoted context omitted.

It is a very common misconception and one of my technical crusades. I keep fighting, but I think I have lost. Not knowing what the "uncertainty interval" represents (is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values?) could be even more dangerous, in theory, than using no uncertainty interval at all. I say in theory because, in my experience in the tech ind…

> is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values If you don't mind typing it out, what do you mean formally here?

I think they mean either what is E[x| y] (standard regression point estimate) along with a confidence interval (this assumes that the mean is a meaningful quantity), or the interval s.t. F(x | y) -- the PDF of x -- is between .025 and .975 (the 95% predictive interval centered around .5). The point is that the width of the confidence interval around the point estimate of the mean converges to 0 as you add more data because you have more information to estimate this point estimate, while the predictive interval does not, it converges to the interval composed of the aleatoric uncertainty of the data generating distribution of x conditioned on the measured covariates y

Re: Forecasts need to have error bars

#84

Post by nonameiguess »

mnky9800n wrote: post #62
bo1024 wrote: post #8

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…

Recently someone on hacker news described statistics as trying to measure how surprised you should be when you are wrong. Big fat error bars would give you the idea that you should expect to be wrong. Skinny ones would highlight that it might be somewhat upsetting to find out you are wrong. I don't think this is an exhaustive description of statistics but I do find it useful when thinking about forecasts.

This can be true but depends on other sources of error being small enough. Standard error is just a formula and it varies inversely with the square of the sample size, so you trivially narrow a confidence interval by sampling more often. In this specific case, imagine you had daily measures of violent crime instead of only annual. You'd get much tighter error bars.

Does that mean you should be more surprised if your predictions are wrong? It depends. You've only reduced model error, but this is the classic precision versus accuracy problem. You can very precisely estimate the wrong number. Does the model really reflect the underlying data generating process? Are the inputs you're giving reliable measurements? If both answers are yes, then your more precise model should be converging toward a better prediction of the true value, but if not, you're only getting a better prediction of the wrong thing.

We can ask these questions with this very example. Clearly, ARIMA is not a causally realistic model. Criminals don't look at last year's crime rates and decide whether to commit a crime based on that. The assumption is that, whatever actually does cause crime, it tends to happen at fairly similar levels year to year, that is, 2020 should be more different than 2010 than it is different from 2019. We may not know what the causually relevant factors really are or we may not be able to measure them, but we at least assume they follow that kind of rule. This sounds plausible to me, but is it true? We can backtest by making predictions of past years and seeing how close they are to the measured value, but the possibility of this even working depends upon the answer to the second question.

So then the second question. Is the national violent crime data actually reliable? I don't know the answer to that, but it certainly isn't perfect. There is a real crime rate for every crime, but it isn't exactly the reported number. Recording and reporting standards vary from jurisdiction to jurisdiction. Many categories of crime go undereported, the extent of which can change over time. Changes may reflect different policing emphasis as much as or more than changes in the underlying true rate. I believe the way the FBI even collects and categorizes data has changed in the past, so I'm not sure a measurement from 1960 can be meaningfully compared to a measurement from 2020.

Ultimately, "how surprised you should be when you are wrong" needs to take all of these sources of error into account, not just the model's coefficient uncertainty.

Re: Forecasts need to have error bars

#85

Post by wayfinder »

nonameiguess wrote: post #84
mnky9800n wrote: post #62

Earlier quoted context omitted.

Recently someone on hacker news described statistics as trying to measure how surprised you should be when you are wrong. Big fat error bars would give you the idea that you should expect to be wrong. Skinny ones would highlight that it might be somewhat upsetting to find out you are wrong. I don't think this is an exhaustive description of statistics but I do find it useful when thinking about forecasts.

This can be true but depends on other sources of error being small enough. Standard error is just a formula and it varies inversely with the square of the sample size, so you trivially narrow a confidence interval by sampling more often. In this specific case, imagine you had daily measures of violent crime instead of only annual. You'd get much tighter error bars. Does that mean you should be more surprised if your…

I disagree.

You only really need to take those sources of error into account if you want an absolute measure of error, which as you explain, seems pretty impossible.

An error for weather only needs to be relative -- for example, if the error for rain today is higher than yesterday, it's not important the exact number higher that it is -- only that it's higher. (Not that I know if this is possible.)

It's like how you can't describe how biased a certain news source is or how to read a Yelp or Rotten Tomatoes review -- you just have to read it often enough to get an intuitive sense that a 4.1 star Yelp-rated restaurant with 800 reviews is probably good while a 4.6 star restaurant with 5 reviews is quite possibly terrible.

Re: Forecasts need to have error bars

#86

Post by bsder »

mightybyte wrote: post #35

Completely agree with this idea. And I would add a corollary...date estimates (i.e. deadlines) should also have error bars. After all, a date is a forecast. If a stakeholder asks for a date, they should also specify what kind of error bars they're looking for. A raw date with no estimate of uncertainty is meaningless. And correspondingly, if an engineer is giving a date to some other stakeholder, they should include…

The problem is that date estimates for deadlines are NOT a standard distribution and everybody's normal statistical tools do not work.

They are pretty much a one sided distribution power law. Deadlines almost never come in early and, when they do, it's rarely by much. On the other hand, deadlines can come in late by wild amounts.

Generating confidence intervals on that is really hard.

Re: Forecasts need to have error bars

#87

Post by Retric »

nonameiguess wrote: post #84
mnky9800n wrote: post #62

Earlier quoted context omitted.

Recently someone on hacker news described statistics as trying to measure how surprised you should be when you are wrong. Big fat error bars would give you the idea that you should expect to be wrong. Skinny ones would highlight that it might be somewhat upsetting to find out you are wrong. I don't think this is an exhaustive description of statistics but I do find it useful when thinking about forecasts.

This can be true but depends on other sources of error being small enough. Standard error is just a formula and it varies inversely with the square of the sample size, so you trivially narrow a confidence interval by sampling more often. In this specific case, imagine you had daily measures of violent crime instead of only annual. You'd get much tighter error bars. Does that mean you should be more surprised if your…

You can arbitrarily scale error bars based on real world feedback, but the underlying purpose of a model is rarely served by such tweaking. Often the point of error bars is less “How surprised you should be when you are wrong” than it is “how wrong you should be before you’re surprised.”

When trying to detect cheating in online games you don’t need to predict exact performance, but you want to decent anomalies quickly. Detecting cereal killers, gang wars, etc isn’t about nailing the number of murders on a given day but patterns within those cases etc.

Re: Forecasts need to have error bars

#89

Post by wizzwizz4 »

Retric wrote: post #87
nonameiguess wrote: post #84

Earlier quoted context omitted.

This can be true but depends on other sources of error being small enough. Standard error is just a formula and it varies inversely with the square of the sample size, so you trivially narrow a confidence interval by sampling more often. In this specific case, imagine you had daily measures of violent crime instead of only annual. You'd get much tighter error bars. Does that mean you should be more surprised if your…

You can arbitrarily scale error bars based on real world feedback, but the underlying purpose of a model is rarely served by such tweaking. Often the point of error bars is less “How surprised you should be when you are wrong” than it is “how wrong you should be before you’re surprised.” When trying to detect cheating in online games you don’t need to predict exact performance, but you want to decent anomalies quickl…

Is this the difference between Bayesian and Frequentist approaches?

Re: Forecasts need to have error bars

#90

Post by bo1024 »

m-murphy wrote: post #83
bo1024 wrote: post #54

Earlier quoted context omitted.

> is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values If you don't mind typing it out, what do you mean formally here?

I think they mean either what is E[x| y] (standard regression point estimate) along with a confidence interval (this assumes that the mean is a meaningful quantity), or the interval s.t. F(x | y) -- the PDF of x -- is between .025 and .975 (the 95% predictive interval centered around .5). The point is that the width of the confidence interval around the point estimate of the mean converges to 0 as you add more data b…

Ah, that makes sense. The word expectation was really throwing me off, along with the fact that, in the kind of forecasting setting of this post, the mean and confidence interval (used in the correct sense) are not meaningful, while the quantile or 'predictive interval' are meaningful.
Post reply on HN