Andrew Gelman designed the 538 model in 2007. Nate Silver authored an adjustment to polls used in that model. Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections, and likely why they were so inaccurate in 2016. Gelman’s own grad student is the only p…
Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
41–50 of 243 posts
Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
#42Earlier quoted context omitted.
How is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after the debacle that was 2016.
They were generally correct with their prediction except in 3 states where nobody had been doing detailed polling because the pollsters didn't think it would matter. It simply wouldn't have been possible for the models to be more accurate with the data they had. As they say, bad data in, bad data out. That's why this time around the pollsters made sure to be more thorough in their polling.
"This time is different."
I've heard that enough times to be highly skeptical. I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results. Cultural and historical trends play a drastically higher role and are almost always left out.
Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
#43Earlier quoted context omitted.
While numerically literate, I don't understand the details of the 538 or economist models. What I do know is that 538's model has had a great track record. It gave Trump one of the highest chances of winning in 2016. It did very well in prior elections. And both models are essentially predicting the same results: ~10% chance of Trump winning.
How is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after the debacle that was 2016.
Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
#44Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
#45I'm not sure why that kind of interstate correlation should impact predictions?
IANAS but it feels like these correlations were added to compensate for the failure in 2016 to recognize that state A going one way implied that state B would also go that way. It "feels" like a more correct approach would be to compute some kind of error/weakness measure in a states polls by bringing in those of its geographical neighbors and incorporating the polling error of that entire block vs prior years. Or something.
The intuition I'm having difficulty conveying is that actual voting correlation is based on neighboring states only because you've got bubbles of ideology that aren't strictly cut along state lines. If strength of opinion in a bubble is going one way, then you'll see that mostly in the state at the center of the bubble, but the bubble still spreads into neighboring states, and a "stronger" bubble could push it geographically further into those neighbouring states, and/or could increase the bias in areas inside the bubble.
[1] "Always" == most recent history
Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
#46Earlier quoted context omitted.
How is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after the debacle that was 2016.
Saying that the ultimate outcome had a 1 in 4 chance is not wrong, slightly wrong, or less wrong. If the weatherman says there's a 1 in 4 chance of rain, and it rains, he wasn't wrong.
Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
#47Every single one of these models will break down this year. We are living in an unprecedented time. I can't understand how we can model how many people will vote, when we don't even know how many people have moved out of cities this year. Half of my friends have left San Francisco - if as many people left Philadelphia, The Twin cities, Milwaukee or Pittsburgh, then that really effects the outcome.
Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
#48The thing about this election is that what if Trump himself is some sort of wildcard that can't really be properly forecasted in the polls? Why is there so much fascination with polls to begin with? I understand that there are betting markets, but it seems sort of silly. If you had a 100% accurate poll, for instance, then what would be the purpose of the actual election?
Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
#49Earlier quoted context omitted.
How is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after the debacle that was 2016.
Saying that the ultimate outcome had a 1 in 4 chance is not wrong, slightly wrong, or less wrong. If the weatherman says there's a 1 in 4 chance of rain, and it rains, he wasn't wrong.
Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast
#50Earlier quoted context omitted.
They said Trump had a 1 in 4 chance. That's very high. NYT had something like 1 in 20 chance for Trump.
That doesn't really answer my question. It only indicates that they were slightly less wrong than every other media source, not that they have a good model. If I had a laptop that only worked 1/4th of the time, rather than 1/20th of the time, would that make it a reliable laptop? I don't think so.
Also, it doesnt make sense to look at a single prediction to evaluate a model.
Out of all the predictions they have made (did you look at individual state predictions?), how many were correct (and how confident were they?) - how many were wrong (and how close to 50% were they?).
That is how you evaluate a model (aka cross entropy)