Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

71–80 of 243 posts

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#71

Post by thaumasiotes »

an_opabinia wrote: post #31

Andrew Gelman designed the 538 model in 2007. Nate Silver authored an adjustment to polls used in that model. Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections, and likely why they were so inaccurate in 2016. Gelman’s own grad student is the only p…

> Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections

I find this argument strange, because black turnout was unusually high in 2008. That should have a negative impact on the accuracy of statistical adjustments, not a positive one.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#72

Post by RivieraKid »

yk wrote: post #52

Somewhat interesting, however the guy lost me more and more the longer he argues. So, the various anomalies in the dataset are somewhat interesting, but having weird outliers in the margins is an entirely expected effect. Just because there are not many datapoints. So when you filter for something marginal like Trump winning New Jersey, then the statistical error increases and therefore it is entirely unsurprising th…

> Just because there are not many datapoints.

There are more than enough data points to determine the between-state error correlations, many of which seem to be very off.

> Additionally, getting worked up about a 3% chance

The weird between-state correlations actually have a large effect, they increase state and nationwide uncertainty and as a result Trump has a higher chance of winning.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#73

Post by RivieraKid »

aakilfernandes wrote: post #14

Could this just be a result of low sampling by the author? If you reduce a sample to only include some tiny edge case, the resulting data points are going to be weird in random ways.

No, there's enough data to determine all between-state error correlations.

Edit: Why the downvote? Each of the between-state correlations can be calculated from 40,000 datapoints.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#74

Post by ceilingcorner »

chki wrote: post #60
ceilingcorner wrote: post #42

Earlier quoted context omitted.

> That's why this time around the pollsters made sure to be more thorough in their polling. "This time is different." I've heard that enough times to be highly skeptical. I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results. Cultural and historical trends play a drastically higher role and are almost always left out.

> I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results. But it has been strongly correlated to the results in basically all elections so far in all democracies on the planet. Taking 2016 as an example there has been a very strong correlation between polling and the results. The national polling averages were only 3 points off from the actual result. If that's not correlat…

Polarization and the unacceptability of publicly saying "I voted for X" also didn't really exist prior to 2016. The fear of getting doxxed, combined with a record low level of trust in institutions and the media, leads to skepticism toward answering polls truthfully, IMO.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#75

Post by wtallis »

Ibethewalrus wrote: post #64
hn_throwaway_99 wrote: post #24

Earlier quoted context omitted.

Are you referring to the 2016 election? If so, you are wrong. 538 gave Trump a higher chance of winning than pretty much every independent pollster.

nope, you're wrong. 538 gave Hillary a 71.4% chance of winning https://projects.fivethirtyeight.com/2016-election-forecast/

The assertion you're responding to was not about Trump's odds in an absolute sense, but relative to other pollsters and forecasters. So pointing out the precise value FiveThirtyEight assigned to that outcome doesn't refute anything until you compare it to somebody else's number.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#76

Post by bart_spoon »

ceilingcorner wrote: post #46
massysett wrote: post #43

Earlier quoted context omitted.

Saying that the ultimate outcome had a 1 in 4 chance is not wrong, slightly wrong, or less wrong. If the weatherman says there's a 1 in 4 chance of rain, and it rains, he wasn't wrong.

No, but it means that the weatherman isn't particularly effective at forecasting the weather.

No it doesn’t, and this is a fundamental misunderstanding of how probabilistic forecasting works. If it rains 9 out of 10 times a weatherman says there is a 30% chance of rain, they are a bad weatherman, but they aren’t much worse than if it rained 0 out of 10 times they predicted a 30% chance of rain. A weatherman accurately assessing the probability of the weather forecast would see it rain around 3 out of 10 days they say there is a 30% chance of rain.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#77

Post by khuey »

ceilingcorner wrote: post #59
khuey wrote: post #54

Earlier quoted context omitted.

This is a fundamental misunderstanding of probability. Low probability events do happen, and it doesn't inherently mean the estimated probability was wrong.

I am criticizing their labeling of Trump's win as low probability.

"He won" is not sufficient evidence upon which to do that.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#78

Post by patrickthebold »

Ibethewalrus wrote: post #64
hn_throwaway_99 wrote: post #24

Earlier quoted context omitted.

Are you referring to the 2016 election? If so, you are wrong. 538 gave Trump a higher chance of winning than pretty much every independent pollster.

nope, you're wrong. 538 gave Hillary a 71.4% chance of winning https://projects.fivethirtyeight.com/2016-election-forecast/

The GP is saying 538 did better compared to other news outlets. You need to also provide a source for another news outlet giving Hillary less than 71.4% if you want to show that the GP is wrong.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#79

Post by ceilingcorner »

khuey wrote: post #77
ceilingcorner wrote: post #59

Earlier quoted context omitted.

I am criticizing their labeling of Trump's win as low probability.

"He won" is not sufficient evidence upon which to do that.

Nowhere did I say it is. I simply think that, if one were following the right information, his win was not as unexpected as the coastal media presented it as being.

Personally I would have put it about 60-40 Hillary-Trump.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#80

Post by untog »

School-Cotton wrote: post #41
an_opabinia wrote: post #31

Andrew Gelman designed the 538 model in 2007. Nate Silver authored an adjustment to polls used in that model. Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections, and likely why they were so inaccurate in 2016. Gelman’s own grad student is the only p…

Nate Silver said Trump had a 1 in 3 chance, which basically means one shouldn’t be surprised no matter the result. I’m not sure where this “all the polls were so far off in 2016!!” narrative comes from, but it’s wrong.

Nate was the outlier in that respect. But it’s true that the polls aren’t weren’t all that inaccurate in 2016: a bunch of important swing states were within the margin of error and Trump won some important states by very small margins.

The mistake in 2016, IMO was a) the extrapolation that came from those polls and b) people paying way too much attention to national polls, which have very little connection to electoral outcomes, given the electoral college.

Also perhaps c) the larger public not “getting” statistics in the way they’ve been presented. The NYT had, if I recall, Clinton at 90% chance of winning. That still means that in one of every ten flips of a coin is a Trump win. But people read “90% chance” as “definite win”. I don’t actually know what anyone should or could do about that.

Post reply on HN