Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

11–20 of 243 posts

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#11

Post by stjohnswarts »

KKKKkkkk1 wrote: post #6

My biggest issue is when people say that it's a probabilistic model, and therefore it wasn't wrong in 2016 because 28% chance of winning is pretty high and you don't get probabilities. Well, guess what, this kind of model that provides a probabilistic estimate on a future event that cannot be repeated cannot be validated or falsified. It's basically junk science (if it has any aspirations of being scientific).

So all statistics and probabilities are junk science because they can't predict the future 100% of the time? Surely you can't be asserting that...

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#12

Post by bena »

Marazan wrote: post #10
KKKKkkkk1 wrote: post #6

My biggest issue is when people say that it's a probabilistic model, and therefore it wasn't wrong in 2016 because 28% chance of winning is pretty high and you don't get probabilities. Well, guess what, this kind of model that provides a probabilistic estimate on a future event that cannot be repeated cannot be validated or falsified. It's basically junk science (if it has any aspirations of being scientific).

If I tell you an unweighted 6 sided die only has a 1 in 6 chance of coming up 6 and we roll it once and it comes up a 6 then that is not junk science.

Yeah, but what if we could never roll that particular die again?

I think that's what he's talking about.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#13

Post by civilized »

KKKKkkkk1 wrote: post #6

My biggest issue is when people say that it's a probabilistic model, and therefore it wasn't wrong in 2016 because 28% chance of winning is pretty high and you don't get probabilities. Well, guess what, this kind of model that provides a probabilistic estimate on a future event that cannot be repeated cannot be validated or falsified. It's basically junk science (if it has any aspirations of being scientific).

Or... if you think for a second about what probabilities are supposed to mean, there is an obvious way to check if a probabilistic forecaster is accurate https://projects.fivethirtyeight.com/checking-our-work/

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#15

Post by cwhiz »

KKKKkkkk1 wrote: post #6

My biggest issue is when people say that it's a probabilistic model, and therefore it wasn't wrong in 2016 because 28% chance of winning is pretty high and you don't get probabilities. Well, guess what, this kind of model that provides a probabilistic estimate on a future event that cannot be repeated cannot be validated or falsified. It's basically junk science (if it has any aspirations of being scientific).

No one is purporting election forecasting to be scientific.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#16

Post by chrisfrantz »

preommr wrote: post #8

> It didn't take very long to do the analysis. But it did then take another hour or so to write it up. It's very interesting to see how long it takes people to do things. I am amazed that entire article took 1 hour to type up. I've spent entire afternoons trying to write shallower pieces of work.

It looks like it was written as a single stream of conscience. While I couldn’t write that article, if I hit a flow state and was interested in the topic, it seems possible.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#17

Post by nullc »

Meh. If you fit a model and don't explicitly constrain against "un-physical" results like negative correlations, you'll end up with them.

Constraining against them won't improve your models fit (usually by definition), and it doesn't always improve robustness (at least for situations near average)-- because they're acting to debias the model in ways that you otherwise don't have enough degrees of freedom to address.

A negative correlation here is also potentially historically supported, in the sense that sometimes DEM/GOP candidates are philosophically reversed in some way relevant to the state. As in, "The only way a GOP would get elected in X is if they had the DEM position on subject Y which would make them lose state Z, who cares as much about that subject as X but in the opposite direction."

Now-- it doesn't seem likely case in this election (e.g. Trump is not (currently) a massively pro-choice republican), so it probably shouldn't apply here-- but it's isn't hard for me to imagine how a negative correlation might show up out of the historical data.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#18

Post by mac01021 »

bena wrote: post #12
Marazan wrote: post #10

Earlier quoted context omitted.

If I tell you an unweighted 6 sided die only has a 1 in 6 chance of coming up 6 and we roll it once and it comes up a 6 then that is not junk science.

Yeah, but what if we could never roll that particular die again? I think that's what he's talking about.

This is just Frequentism vs Bayesianism, right?

https://en.wikipedia.org/wiki/Probability_interpretations

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#19

Post by NhanH »

bena wrote: post #12
Marazan wrote: post #10

Earlier quoted context omitted.

If I tell you an unweighted 6 sided die only has a 1 in 6 chance of coming up 6 and we roll it once and it comes up a 6 then that is not junk science.

Yeah, but what if we could never roll that particular die again? I think that's what he's talking about.

You treat the forecaster as the dice, not a particular forecast from them. You can measure their forecasts with reality.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#20

Post by HeavenFox »

KKKKkkkk1 wrote: post #6

My biggest issue is when people say that it's a probabilistic model, and therefore it wasn't wrong in 2016 because 28% chance of winning is pretty high and you don't get probabilities. Well, guess what, this kind of model that provides a probabilistic estimate on a future event that cannot be repeated cannot be validated or falsified. It's basically junk science (if it has any aspirations of being scientific).

This is basically the frequentist vs bayesian debate. Looks like you are firmly in the former camp :)
Post reply on HN