Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast • Hacker News Board

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#121

Post by evgen » Sun, Oct 25, 2020, 5:41 PM UTC

justinzollars wrote: post #93

ceilingcorner wrote: post #47

Earlier quoted context omitted.

I've made similar points on topics like this, and bar none, every single time, it is downvoted into oblivion. People seem to have a difficult time with forecasting data that goes against their preferred outcomes.

I'm not trying to get points, or keep points. I'm just trying to point out that the models are not going to work this year and its crazy to assume they will.

The models will work just fine. We are talking about large numbers here, and your small sample of anecdotes does not mean anything. No, there is no mass exodus. No, it is not going to change the results. Yes, the current polls are more accurate than just about any in history and we have an abundance of high-quality state level polls to back up the predictions.

If you want to know where the models are going to break down it is more likely that the state-level polling has over-corrected for the factors that caused them to miss the swing to Trump in 2016 and Biden's numbers are even better than what polls are saying (a prediction based on looking at polling at the congressional district level and then seeing how that differs from state-level polling -- the numbers are the district level are closer to national numbers than the lower statewide numbers for swing states.)

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#122

Post by ceilingcorner » Sun, Oct 25, 2020, 5:42 PM UTC

wtallis wrote: post #118

ceilingcorner wrote: post #108

Earlier quoted context omitted.

I called it a debacle because 99.9% of media sources, pundits, politicians, political figures, or anyone else thought that Hillary had anything less than a guaranteed win. I'm simply suggesting it was actually always a close race, but that the media ignored this because it went against their ideological model / they weren't familiar with places like the Rust Belt.

You really need to be more clear about which of your criticisms are against FiveThirtyEight specifically vs against the media in general. Because now it's looking like you are complaining about the media in general and using that as justification for mistrusting FiveThirtyEight, in the context of a discussion specifically about FiveThirtyEight being a notable outlier from that general media trend.

It's still just unclear to me how FiveThirtyEight assigning Trump a ±30% chance of winning can be considered "accurate" or "good".

From my point of view, this is only because said people considered Trump winning so extremely unlikely that 538 getting it sort of right appears exceptional. In reality, they were still quite wrong, just slightly less so. Ergo I don't see much value in their model.

I'm seeing the same exact thing today with Biden at a 90% chance of winning.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#123

Post by shajznnckfke » Sun, Oct 25, 2020, 5:43 PM UTC

ceilingcorner wrote: post #86

shajznnckfke wrote: post #82

Earlier quoted context omitted.

If you have a die that rolls a one 1/6 of the time, do you consider the die wrong?

No, but I don't consider it useful toward predicting the outcome, which is of course what this is all about.

If there was sufficient data to assign a 0% or 100% probability to an event, that’s what a forecaster should do. If there isn’t sufficient data, then anyone who claims there is a sure thing is a charlatan.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#124

Post by DavidSJ » Sun, Oct 25, 2020, 5:45 PM UTC

RivieraKid wrote: post #84

olliej wrote: post #45

It seems like the behavior between WA and MS could just be statistics saying that WA and MS always[1] vote for the opposite candidate, rather than considering a massive sudden change in the direction that one of them votes in. E.g. it's not reflecting who they vote, just who they most vehemently disagree with. I'm not sure why that kind of interstate correlation should impact predictions? IANAS but it feels like thes…

> I'm not sure why that kind of interstate correlation should impact predictions? 538 has low positive correlations between states on average, which actually has a big impact, it increases overall uncertainty (and therefore Trump's win probability). Why? If the states are not correlated, you usually end up with a few states going off the rails, like Trump winning Colorado without any nationwide swing.

Other way around: uncorrelated errors tend to cancel each other, correlated errors tend to reinforce each other.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#125

Post by noirbot » Sun, Oct 25, 2020, 5:45 PM UTC

RivieraKid wrote: post #105

nullc wrote: post #17

Meh. If you fit a model and don't explicitly constrain against "un-physical" results like negative correlations, you'll end up with them. Constraining against them won't improve your models fit (usually by definition), and it doesn't always improve robustness (at least for situations near average)-- because they're acting to debias the model in ways that you otherwise don't have enough degrees of freedom to address.…

> If you fit a model and don't explicitly constrain against "un-physical" results like negative correlations, you'll end up with them. The Economist model does exactly that, and all of their correlations are positive. I recommend reading their methodology, they know what they're doing (I wouldn't say the same about 538). Andrew Gelman has developed some of the Bayesian methods and software that people like Nate Silve…

I think the question is if it matters to the predictive accuracy of the model. Just because it puts out results you can't envision actually happening on the margins doesn't mean they can't happen, or that they can't be valuable in presenting a holistic result.

It's clear that the models are tuned differently, but from Silver's replies in the PS's, it seems that he's ok with these artifacts being part of the model.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#126

Post by peteradio » Sun, Oct 25, 2020, 5:47 PM UTC

The problem is that 538 is correctly factoring the voting fuckery done at the state level which ruins the voting correlations. Gelman seems to be modelling fair elections - ha! Now the question, how did 538 come up with the correct model which takes into account vote manipulations at the state level? /s

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#127

Post by evgen » Sun, Oct 25, 2020, 5:47 PM UTC

Traster wrote: post #90

> I’d think that if Trump were to win New Jersey or, even more so, California, that this would most likely happen only as part of a national landslide of the sort envisioned by Scott Adams or whatever. That's a valid intuition to have but you can also clearly make the argument that if Trump wins California you're in such a weird scenario that using the traditional wisdom about correlation is dangerous. The point that…

This is the important point here IMHO. There are two errors that the tails need to deal with, voter shifts that are missed by polling and black swan events that completely upend the table. I think that the 528 model lumps a lot of the long tail into the second category, which then basically becomes a "let's throw out most of the rules and make wild guesses" territory. There is so little worthwhile information in those tails I am really surprised that this is the focus of the disagreement.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#128

Post by drblast » Sun, Oct 25, 2020, 5:49 PM UTC

noirbot wrote: post #89

To me, this mostly tracks with what 538 has said on record about how their model works and the design philosophy behind parts of it. To me, what Nate means when he says "directionally the right approach in terms of our model's takeaways" is that these sorts of wild and unintuitive outcomes are part of the point of the way the model is constructed. Specifically, that when you get off into the weird situations like Tru…

Yeah I think for Trump to win washington state, he'd have to do something to appeal to voters there in a way that would likely cause his red state base to abandon him.

The negative correlation makes sense when we think about how difficult it is for everyone in Washington to suddenly turn conservative and everyone in Mississippi to turn liberal. Much more likely is that the crazy thing is that the candidate or circumstances changed in some way.

It makes more sense if we ask...if a candidate wins NJ what is the chance they also won AK?

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#129

Post by thomasmeeks » Sun, Oct 25, 2020, 5:53 PM UTC

ceilingcorner wrote: post #86

shajznnckfke wrote: post #82

Earlier quoted context omitted.

If you have a die that rolls a one 1/6 of the time, do you consider the die wrong?

No, but I don't consider it useful toward predicting the outcome, which is of course what this is all about.

Say more about "but I don't consider it useful toward predicting the outcome"

How else would one predict the outcome of a die roll, specifically?

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#130

Post by wtallis » Sun, Oct 25, 2020, 5:53 PM UTC

an_opabinia wrote: post #120

thaumasiotes wrote: post #71

Earlier quoted context omitted.

> Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections I find this argument strange, because black turnout was unusually high in 2008. That should have a negative impact on the accuracy of statistical adjustments, not a positive one.

I think he made an estimate for the increase in black turnout. If I were designing the model, and I believed turnout is the biggest factor (maybe inconclusive among political scientists), I would look at the circumstances where turnout changes based on candidate's demographics and validate it across statewide and congressional races. However, we will never know, because they never published the code.

> I think he made an estimate for the increase in black turnout.

I think that kind of adjustment is usually the responsibility of the pollsters, with their likely voter models. I don't think FiveThirtyEight directly tries to also apply such an adjustment, because that would be at serious risk of overcorrecting.

Similarly, this year many pollsters have added level of education as a factor to their demographic weighting, to address a shortcoming in their 2016 performance. FiveThirtyEight consumes those poll numbers without adding their own layer of demographic adjustment.