Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

If you'd like to not be downvoted into oblivion you'd do well to provide an alternate theory. What's your prediction for the upcoming election? Why? What's your methodology? Or is your point that forecasting is futile? I can't tell. There's too much emotion directed at 538/etc. for me to suss out what your point is besides disliking 538/the media/etc, which just isn't particularly helpful for the discussion.

I think the third post in that thread really nails the underlying issue here. We, as humans, know that it would be silly to think that some configurations could happen. We know there's no reason to expect Biden to win Alabama outside of him winning nearly every state.

A statistical model only has a vague idea of context/the real world. It looks at polls (and probably not really that many polls of Alabama or Mississippi or Alaska) and sees that, statistically, Biden should win 3% of the time or so.

It doesn't have a specific world set of events in mind that would cause that, it just knows that that's how the numbers go, and thus may lead to weird circumstances in the grander results because it has to make the world match the numbers in these small corners.

I wish it were that simple. HN just seems to have gone the way of reddit, where downvoting is disagreement. And of course, 3 minutes after I post this comment, it's downvoted.

I don't have a methodology, because I'm not a pollster with dozens of people at my disposal. I am just bemused and annoyed that things like 538 continue to be taken seriously when they continue to ignore sociological, historical, and cultural factors in favor of an overly-complex quantitative model.

Re: the upcoming election. I don't think we can be sure, yet. Certainly it will be close, and the Biden at 90% to win estimations make no sense to me. Biden is a much weaker candidate than Hillary and he continues to make blunders (i.e. I guarantee that his comments on fracking in the last debate just lost him Pennsylvania.) Trump seems to be finding a lot of allies in strange places, e.g. African-American celebrities. That may be an isolated incident, or it may signal some big unexpected changes.

At this point my estimation is Trump-Biden 55-45, for the simple reason that people tend to vote for economic issues and Trump has a better "perception" on this issue. "It's the economy, stupid." as James Carville put it.

One thing to note: 538 uses a t-distribution (and calls out on regularly in their podcast that this yields much heavier tails than a normal distribution). Even 40,000 samples is not enough to characterize the tails.

So, it seems to me that the entire article is predicated on a faulty conjecture, namely that 538 uses a mixture of a normal distribution with an independent heavy-tailed one. (It's not explicitly stated what the author thinks the base model is, but I think "normal" is a reasonable guess.)

I'd be interested in seeing a reverse-engineering analysis of 538's choice of distribution parameters, and extrapolation from there to see if these pathologies still arise with (much) larger samples.

...

That said, ultimately, the choice of how fat to make the tails is a modeling decision, and how the models behave outside the regime of interest isn't as important as how they behave within the operating region. There are key ways we can evaluate goodness of fit once we have results (e.g. bias, MSE) which we can use to determine just how wrong the model was as a predictor, and chances are pretty good that we won't see, say, Trump winning NJ, so we won't actually be able to validate the tail correlation with the vote in PA. But we will be able to validate the correlation in margin between PA and NJ.

Maybe 538's tails are too fat, and every prediction in the 80-95% range ends up going as predicted. Or maybe they're not fat enough, and some races in the 99% bucket end up going the opposite way. Point is, we won't know for sure which models were the best predictors until we can verify the predictions.

(see: all models are wrong, etc. Newtonian mechanics work great as long as your objects are big and slow, for instance.)

Yes, I'm not defending the original guy, I'm just stating what I believe his argument to be.

He's basically saying he doesn't believe the die is fair/unweighted.

So stating the odds of a fair die is kind of immaterial to his point. We need to demonstrate to the guy that the die is fair.

Someone else posted a link (I think) of 538 going over on how accurate they were. Whether their odds bore out. Here it is: https://projects.fivethirtyeight.com/checking-our-work/

Basically what they did was bucket every prediction by odds. If they predicted 70/30, it went in that bucket. And they're "right" at about the rate of their predictions. In other words, for every 70/30 prediction they made, the people/teams with 70% chance to win, won about 70% of the time.

That shows that 538 in this case is pretty decent at calculating odds.