Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

201–210 of 243 posts

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#201

Post by runawaybottle »

dsaavy wrote: post #178
an_opabinia wrote: post #31

Andrew Gelman designed the 538 model in 2007. Nate Silver authored an adjustment to polls used in that model. Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections, and likely why they were so inaccurate in 2016. Gelman’s own grad student is the only p…

I think polls being more representative of turnout amongst minorities could help indicate a potential black swan event for the election. If turnout does return to 2008 and 2012 election levels, polls featured in this fivethirtyeight article [1] indicate Trump is performing better amongst black and hispanic voters. Both demographics are seeing a 10-15% swing in support for Trump compared to 2016, which could theoretic…

It wouldn’t be a black swan event, it would simply be a variable that got re-toggled on. As in, there was a large black turn out for Obama, and there wasn’t for Hilary. What if we turned that variable back on to true for Biden? That’s about all the rocket science involved.

We’ve seen that variable before.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#202

Post by wtallis »

pessimizer wrote: post #199
sequoia wrote: post #175

As a US voter I am very frustrated With and tired of this obsession with election forecasting. Is it going to influence whether or not you go out and vote? If not, what is the point of it? What value does something like fivethirtyeight add to our democracy, if any? Is this motivation the same as that of diving deep into baseball stats or Star Wars starship engineering, just like “nerding out” for its own sake? Contra…

There's nothing scientific about any of this trash. It's a weird conflation of the degree of the incompetence of pollsters and the degree to which opinions can be changed within a span of time . And it's completely unfalsifiable. When Trump wins, the true believers will say "We gave him a 9.684% chance of winning! It's only your ignorance that makes you think we were wrong" and they go back to poring over their race…

> And it's completely unfalsifiable. When Trump wins, the true believers will say "We gave him a 9.684% chance of winning! It's only your ignorance that makes you think we were wrong" and they go back to poring over their race tables.

> It's an orgy of false precision.

The false precision is pretty obviously coming from you, not the FiveThirtyEight pages that never show more than two (or rarely three) significant figures, and emphasize in every other way they can that the numbers are approximate and uncertain. Have you seen the width of the 80% confidence intervals on their graphs?

As for falsification: all of their predictions are for testable outcomes. We'll always know soon enough who actually wins an election, and which states they won, and by what margin, and who turned out to vote. That's all public record. The only part of the post-hoc analysis that is non-trivial is figuring out how a candidate fared with specific demographic groups. It's imperfect, but between exit polling and precinct-level demographic information and election results, it certainly is possible to detect large pre-election polling errors resulting from inaccurate demographic weighting.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#203

Post by [deleted] »

rsynnott wrote: post #192
ceilingcorner wrote: post #136

Earlier quoted context omitted.

I wish it were that simple. HN just seems to have gone the way of reddit, where downvoting is disagreement. And of course, 3 minutes after I post this comment, it's downvoted. I don't have a methodology, because I'm not a pollster with dozens of people at my disposal. I am just bemused and annoyed that things like 538 continue to be taken seriously when they continue to ignore sociological, historical, and cultural f…

> Biden is a much weaker candidate than Hillary and he continues to make blunders That’s your view; however, he is polling much better than Clinton, which would indicate that voters don’t necessarily agree with you (or else just that peoples’ opinions of Trump are lower than last time round, or a combination. But really it hardly matters). > (i.e. I guarantee that his comments on fracking in the last debate just lost…

[deleted]

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#204

Post by hn_throwaway_99 »

disown wrote: post #133
hn_throwaway_99 wrote: post #24

Earlier quoted context omitted.

Are you referring to the 2016 election? If so, you are wrong. 538 gave Trump a higher chance of winning than pretty much every independent pollster.

So they were "very very" wrong rather than "very very very" wrong like other pollsters? I think you proved the OP right rather than wrong.

Only because people don't understand probability and statistics. 538 gave Trump a ~30% chance of winning. The fact the people seem to think anything less than 50 equals 0 is a problem with people's understanding of statistics, not the statistic itself.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#205

Post by taeric »

i_love_limes wrote: post #177
preommr wrote: post #8

> It didn't take very long to do the analysis. But it did then take another hour or so to write it up. It's very interesting to see how long it takes people to do things. I am amazed that entire article took 1 hour to type up. I've spent entire afternoons trying to write shallower pieces of work.

I think a lot of people (myself included when I've felt the pressure to) lowball how long things like this take because a) it makes me look smart and b) people could judge of they knew who much time I actually wasted on it.

I think you read the parent in the opposite direction than intended. It was praise for getting this done so quickly, by my read.

Granted, i could just be misreading this post. :)

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#206

Post by taeric »

ta8645 wrote: post #115
cwhiz wrote: post #28

Not sure if it’s wrong to put this here, but here is a link to their election forecast. https://projects.economist.com/us-2020-forecast/president You can compare this to the 538 model and see where these two teams and forecasts disagree.

Can someone help me understand what odds like this mean in the context of an election? The model says that Trump has a 1 in 10 chance of winning. With a fair 10-sided die it makes sense that you have a 1 in 10 chance of any given side rolling face up. But what is the die that is being rolled in these election statistics? What is the "chance" element that is being predicted?

It means if you saw all of these facts in ten different events, you would not be surprised to see a one to nine split in results. Ish. As you are scaling up, if course.

So, think of it as saying these facts basically describe a ten sided die. With no other knowledge, the best you have is that you expect it to behave the same as any other ten sided die.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#207

Post by runawaybottle »

TTPrograms wrote: post #185
artichokes wrote: post #172

Earlier quoted context omitted.

I don't know if I'm reading your comment wrong but I would happily take that bet- I think most people would. Want to make it? The election result of California isn't a matter of probability; it's an empirical matter that the number of people who will vote for Biden in California far exceeds the number that will vote for Trump.

To you and the other comment - as much as I dislike Taleb's rhetoric, this is precisely the sort of bet he has made a lot of money on. People round rare event probability down to zero, and if you bet against them enough with sufficiently extreme odds you'll eventually (and in expectation) hit a home run. I would be more than happy to make this bet with anyone willing to take the other side - as in literally, find a m…

But would Taleb specifically make a one time bet on one particular Black swan? Isn’t the idea the same as venture capitalism, where this idea only works if you do it with all possible black swans?

You’d have to be crazy to take this one specific bet, you can only realistically take all possible improbable bets.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#208

Post by dahart »

sequoia wrote: post #175

As a US voter I am very frustrated With and tired of this obsession with election forecasting. Is it going to influence whether or not you go out and vote? If not, what is the point of it? What value does something like fivethirtyeight add to our democracy, if any? Is this motivation the same as that of diving deep into baseball stats or Star Wars starship engineering, just like “nerding out” for its own sake? Contra…

> what is the point of it?

Great question, and I share some of your concern, though I can imagine some positive framings in addition to what you wrote. For example, to use an analogy, what’s the point of trying to predict the weather, or trying to predict the stock market? There are lots of reasons including planning ahead for likely outcomes, the ability to protect against losses, and last but not least making money.

I can also imagine that the desire to talk about the potential outcomes is valuable as a social activity, and doesn’t necessarily need to meet a standard of influencing the vote, or adding to our democracy.

> My concern is that these things are distracting and may actually dissuade some people from voting because they think they “don’t have to.”

Of course if your concern is founded, this can go both ways... if the polls show the candidate you favor starting to lose, it could be a call to vote.

If polls are distracting and dissuade voters, then unfortunately election results might do exactly the same or worse. When a state has been solidly red or blue and not purple for 50 years in a row, people do (perhaps rightly so) jump to conclusions about the outcome in advance.

One question we could ask is whether, if voting were made mandatory, would election predictions go away? I’d speculate no.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#209

Post by runawaybottle »

klyrs wrote: post #63

The negative correlation between NJ and AK is curious. What I'd like to see here (and in all of these forecast sites) is some confidence analysis. If you pick Trump in NJ, looking at the plots, you've selected a tiny fraction of the data to examine. Who cares if the predicted value is wonky; show me the confidence interval! Treating this as a tea-leaf reading (that is, deliberately searching for meaning via free asso…

This is the 2008 election result county map in NY:

https://upload.wikimedia.org/wikipedia/commons/thumb/1/1f/Ne...

This is it in 2016:

https://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Ne...

Long Island is really red there. It’s really hard to say how a democratic stronghold like NYC and something literally a 45 min train ride next to it could vote so differently. Long Islanders are not separate from NYCers, they commute to and work in the city.

To your question, could experiments work in similar situations like this across the country for either side? I think so in the next 50 years as demographics shift (and I don’t think it’s as simple as urban liberals taking over, people do become more conservative as they get older). God knows the dynamic at work between NYC and Long Island in 2016, but it’s obvious things are in flux.

I’ll make a bold prediction here. If Long Island is that red again, yeah, you better believe the typical rust belt states are staying red.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#210

Post by comex »

runawaybottle wrote: post #187
aakilfernandes wrote: post #148

Earlier quoted context omitted.

You're comparing two scenarios, one in which you know all the facts, and one in which you don't. In the dice toss scenario, we know everything relevant. In the election scenario, we don't. A model like this is attempting to say "these are the rules we think exist. Based on the rules, and assuming the data is off by some random distribution, here's what we think could happen". What different forecasters disagree about…

I will veer this off into the dreaded political territory even though this is mostly a technical discussion. The Democratic Party proved it was not as progressive as they thought as Sanders lost the primary. The reality is, the country as a whole is also not as liberal either, regardless of what these pollsters are asking people. You think the party is youthful, and ready for progressive ideas, but alas, the party wh…

> The Democratic Party proved it was not as progressive as they thought as Sanders lost the primary.

The FiveThirtyEight forecast for the Democratic primary [1] gave Biden the highest chance of winning for most of the process. He did have a steep drop in the month before Super Tuesday (followed by an equally steep rebound), but still, I wouldn't say the forecast was especially bad. That said, polling is always worse for primaries than general elections, since there are more candidates and fewer voters.

[1] https://projects.fivethirtyeight.com/2020-primary-forecast/

Post reply on HN