Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast • Hacker News Board

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#51

Post by jacquesm » Sun, Oct 25, 2020, 5:08 PM UTC

chrisfrantz wrote: post #16

preommr wrote: post #8

> It didn't take very long to do the analysis. But it did then take another hour or so to write it up. It's very interesting to see how long it takes people to do things. I am amazed that entire article took 1 hour to type up. I've spent entire afternoons trying to write shallower pieces of work.

It looks like it was written as a single stream of conscience. While I couldn’t write that article, if I hit a flow state and was interested in the topic, it seems possible.

The trick is to think before you write. The same goes for programming. If you already know what you are going to write or build then you can reach very high apparent productivity, the time spent on thinking about it isn't accounted for.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#52

Post by yk » Sun, Oct 25, 2020, 5:08 PM UTC

Somewhat interesting, however the guy lost me more and more the longer he argues. So, the various anomalies in the dataset are somewhat interesting, but having weird outliers in the margins is an entirely expected effect. Just because there are not many datapoints. So when you filter for something marginal like Trump winning New Jersey, then the statistical error increases and therefore it is entirely unsurprising that something weird happens. Thankfully, these systems are designed to work with probabilities, and these outliers are weighted down.

Additionally, getting worked up about a 3% chance of Biden winning Alabama. I mean, what does a 3% chance even mean for a one off event, compared to a 5% chance or a .3% chance? I know fully well, that it means I should bet $100 if I can get more than $3000 payout, but the trouble is that is only if we bet often enough. (Perhaps often enough on different things.) For a one off thing, the important part is, it is with a very high degree of certainty a loss of $100. So any claims that Bidens chances of winning are too high should be regarded with high suspicion.

Also, I listened eralier to Nate Silver's model talk [0], where he discusses quite a few problems with low quality polls in some states.

[0] https://fivethirtyeight.com/features/politics-podcast-nation...

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#53

Post by ceilingcorner » Sun, Oct 25, 2020, 5:09 PM UTC

delecti wrote: post #40

ceilingcorner wrote: post #36

Earlier quoted context omitted.

That doesn't really answer my question. It only indicates that they were slightly less wrong than every other media source, not that they have a good model. If I had a laptop that only worked 1/4th of the time, rather than 1/20th of the time, would that make it a reliable laptop? I don't think so.

It's not "wrong" to predict a low chance for something that eventually happens. Unlikely events can happen.

That isn't the criticism. The criticism is the appellation of it being "unlikely."

For example: anyone paying attention to the Rust Belt ±1980-2016 would have dramatically upped Trump's chances in Pennsylvania and Michigan. FiveThirtyEight had Hillary with 70%+ chance of winning both, which to me, shows a deep ignorance of actual cultural factors.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#54

Post by khuey » Sun, Oct 25, 2020, 5:09 PM UTC

ceilingcorner wrote: post #34

abnry wrote: post #33

Earlier quoted context omitted.

While numerically literate, I don't understand the details of the 538 or economist models. What I do know is that 538's model has had a great track record. It gave Trump one of the highest chances of winning in 2016. It did very well in prior elections. And both models are essentially predicting the same results: ~10% chance of Trump winning.

How is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after the debacle that was 2016.

This is a fundamental misunderstanding of probability. Low probability events do happen, and it doesn't inherently mean the estimated probability was wrong.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#55

Post by Tyr42 » Sun, Oct 25, 2020, 5:09 PM UTC

ceilingcorner wrote: post #36

abnry wrote: post #35

Earlier quoted context omitted.

They said Trump had a 1 in 4 chance. That's very high. NYT had something like 1 in 20 chance for Trump.

That doesn't really answer my question. It only indicates that they were slightly less wrong than every other media source, not that they have a good model. If I had a laptop that only worked 1/4th of the time, rather than 1/20th of the time, would that make it a reliable laptop? I don't think so.

If I tell you you're not likely to get two heads in a row, and you do, does that make me un-reliable?

It's unfortunate we can't just run the election again a few times, and actually find the rate at which Trump is elected given the polls.

And it's not empty signalling if 538 assigned Trump a higher chance of winning; they were pretty much the only ones saying he has a chance. That is why people think the models are useful.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#56

Post by vanattab » Sun, Oct 25, 2020, 5:09 PM UTC

School-Cotton wrote: post #41

an_opabinia wrote: post #31

Andrew Gelman designed the 538 model in 2007. Nate Silver authored an adjustment to polls used in that model. Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections, and likely why they were so inaccurate in 2016. Gelman’s own grad student is the only p…

Nate Silver said Trump had a 1 in 3 chance, which basically means one shouldn’t be surprised no matter the result. I’m not sure where this “all the polls were so far off in 2016!!” narrative comes from, but it’s wrong.

The narrative comes from the medias inaccurate and misleading coverage of the polls in 2016. Many news outlets all but declared Clinton president before the election.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#57

Post by noirbot » Sun, Oct 25, 2020, 5:10 PM UTC

ceilingcorner wrote: post #46

massysett wrote: post #43

Earlier quoted context omitted.

Saying that the ultimate outcome had a 1 in 4 chance is not wrong, slightly wrong, or less wrong. If the weatherman says there's a 1 in 4 chance of rain, and it rains, he wasn't wrong.

No, but it means that the weatherman isn't particularly effective at forecasting the weather.

Only if they say that every day, and it rains every day. If the most likely outcome happened every time, then the model is likely wrong/under-confident. The prediction is never going to be 100% accurate until the event is happening/has happened. Up until then, there's always a chance you're wrong or something can change. Being wrong once isn't necessarily a sign that the whole system is messed up.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#58

Post by jsnell » Sun, Oct 25, 2020, 5:11 PM UTC

chki wrote: post #30

happytoexplain wrote: post #22

Earlier quoted context omitted.

That's a pretty bold claim to make about essentially anybody in regards to the US presidential election. Not that I don't believe his account is even-handed and valuable, just that I'm curious what makes you say he has no dog in the fight.

I think he is not talking about the election in general but referencing the "fight" between the 538 model/Nate Silver vs. the Economist/Andre Gelman.

Correct.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#59

Post by ceilingcorner » Sun, Oct 25, 2020, 5:11 PM UTC

khuey wrote: post #54

ceilingcorner wrote: post #34

Earlier quoted context omitted.

How is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after the debacle that was 2016.

This is a fundamental misunderstanding of probability. Low probability events do happen, and it doesn't inherently mean the estimated probability was wrong.

I am criticizing their labeling of Trump's win as low probability.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#60

Post by chki » Sun, Oct 25, 2020, 5:12 PM UTC

ceilingcorner wrote: post #42

gamblor956 wrote: post #38

Earlier quoted context omitted.

They were generally correct with their prediction except in 3 states where nobody had been doing detailed polling because the pollsters didn't think it would matter. It simply wouldn't have been possible for the models to be more accurate with the data they had. As they say, bad data in, bad data out. That's why this time around the pollsters made sure to be more thorough in their polling.

> That's why this time around the pollsters made sure to be more thorough in their polling. "This time is different." I've heard that enough times to be highly skeptical. I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results. Cultural and historical trends play a drastically higher role and are almost always left out.

> I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results.

But it has been strongly correlated to the results in basically all elections so far in all democracies on the planet. Taking 2016 as an example there has been a very strong correlation between polling and the results. The national polling averages were only 3 points off from the actual result. If that's not correlated I don't know what is.