Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forecasts need to have error bars

andrewpwheeler.com

Post reply on HN
Print view View original

101–110 of 165 posts

Re: Forecasts need to have error bars

#101

Post by eru »

bo1024 wrote: post #79
PeterisP wrote: post #63

Earlier quoted context omitted.

There are ways of scoring forecasts that reward accurate-and-certain forecasts in a manner where it's provably optimal to provide the most accurate estimates for your (un)certainty as you can.

Yes, of course. I don't see that as very related to my point. For example, consider how 538 or The Economist predict elections. They might claim they'll use squared error or log score, but when it comes down to a big mistake, they'll blame it on factors outside their models.

Well, but at least 538 has a reputation to defend as an accurate forecaster. So they have some skin in the game.

(Of course, that's not as good as betting money.)

Re: Forecasts need to have error bars

#102

Post by eru »

hgomersall wrote: post #58
bo1024 wrote: post #56

Earlier quoted context omitted.

To me, this comes back to the question of skin in the game. If you have skin in the game, then you produce the best uncertainty estimates you can (by any means). If you don't, you just sit back and say "well these are the error bars my model came up with".

It's worse than that. Oftentimes the skin in the game provides a motivation to mislead. C.f. most of the economics profession.

How do economists have skin in the game?

Many of them eg work in universities and some even have tenure. There's not much skin in the game between any forecasts they might make and their academic prospects.

Economists working for companies often have to help them understand micro and macro-economics. Eg (some of) Google's economists help them design the ad auctions. It's relatively easy to figure out for Google how well those ad auctions work. So they certain have skin in the game. But: what motivation to mislead do those economists have?

Re: Forecasts need to have error bars

#103

Post by Arrath »

I'm just imagining adding error bars to my schedule forecasting (with schedules that are typically one the optimistic side thanks to management), with bars pointing in the bad direction, and seeing management still insist it'll take too long.

Re: Forecasts need to have error bars

#105

Post by mightybyte »

niebeendend wrote: post #47
mightybyte wrote: post #35

Completely agree with this idea. And I would add a corollary...date estimates (i.e. deadlines) should also have error bars. After all, a date is a forecast. If a stakeholder asks for a date, they should also specify what kind of error bars they're looking for. A raw date with no estimate of uncertainty is meaningless. And correspondingly, if an engineer is giving a date to some other stakeholder, they should include…

A deadline implies the upper limit of error bar cannot exceed it. That means you need to appropriately buffer to hit the deadline.

I don't think that's the way it works out in practice. The fact of the matter is that deadlines are missed all the time. In many cases, there is no such thing as 100% certainty that you'll hit a "deadline"--there are always circumstances outside your control (global pandemics anyone?). There's just some implicit confidence threshold or other assumptions lurking around that probably need to be communicated. Do you want three 9s of confidence? Five 9s? Those things are very different and the cost to actually achieve the latter can often be prohibitive. Everyone benefits if we make explicit our pre-conceived idea of precisely what "cannot exceed" means.

Re: Forecasts need to have error bars

#106

Post by jrumbut »

eru wrote: post #100
ISL wrote: post #99

Earlier quoted context omitted.

A position espoused by Bill Phillips [1], and to which I now adhere: "You should be willing to take either side of the bet that confidence interval implies." (paraphrasing; he says it better). For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval. Aside fro…

> Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course) Also assuming normal distribution, I think? > If the notion of letting your reader take either side of the bet makes your stomach a little queasy, you're on the right track. The feeling will subside when you're pretty sure yo…

> Also assuming normal distribution, I think?

95% is 95% regardless of the distribution.

> I would like to build some edge into my bets. If a reader takes both sides of your example, they would be come out exactly even.

You can imagine yourself being equally unhappy to take either side of the bet, if that's easier than imagining yourself being happy to take either side.

It is for me, which is probably something to bring up in therapy.

I also think that framing things as bets brings in all the cultural baggage around gambling and so it isn't always helpful. I'm not sure what a better framing is though.

Re: Forecasts need to have error bars

#107

Post by staunton »

mightybyte wrote: post #105
niebeendend wrote: post #47

Earlier quoted context omitted.

A deadline implies the upper limit of error bar cannot exceed it. That means you need to appropriately buffer to hit the deadline.

I don't think that's the way it works out in practice. The fact of the matter is that deadlines are missed all the time. In many cases, there is no such thing as 100% certainty that you'll hit a "deadline"--there are always circumstances outside your control (global pandemics anyone?). There's just some implicit confidence threshold or other assumptions lurking around that probably need to be communicated. Do you wan…

Going even further, deadlines are often a tool for signaling "the organization is trying really hard to achieve this fast". The shorter the deadline (as long as it's at least somewhat in theory plausible), the harder you're trying. Often most people involved (even those deciding on the date for the deadline) know from the very start that the deadline will almost certainly be missed.

I regularly see two kinds of deadlines. "Planning deadlines" describe an estimate when something will be done. "Signalling deadlines" signal priorities and motivation to employees or clients. Sometimes both exist in parallel for the same task and there is a subset of people who know both.

Re: Forecasts need to have error bars

#108

Post by staunton »

marcinzm wrote: post #55
nradov wrote: post #31

Earlier quoted context omitted.

Mostly not. Very few data "scientists" working in industry actually follow the scientific method. Instead they just mess around with various statistical techniques (including AI/ML) until they get a result that management likes.

Most decent companies and especially tech do AB testing for everything including having people whose only job is to make sure those test results are statistically valid.

The magic words here are make sure.

Re: Forecasts need to have error bars

#109

Post by gyrovagueGeist »

hgomersall wrote: post #42
bo1024 wrote: post #8

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "erro…

Error bars in forecasts can only mean uncertainty your model has. Without error bars over models, you can say nothing about how good your model is. Even with them, your hypermodel may be inadequate.

They can also mean pushed forward uncertainty from input parameters which isn't exactly the same as model error
Post reply on HN