Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast • Hacker News Board

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#221

Post by Closi » Mon, Oct 26, 2020, 8:22 AM UTC

runawaybottle wrote: post #187

aakilfernandes wrote: post #148

Earlier quoted context omitted.

You're comparing two scenarios, one in which you know all the facts, and one in which you don't. In the dice toss scenario, we know everything relevant. In the election scenario, we don't. A model like this is attempting to say "these are the rules we think exist. Based on the rules, and assuming the data is off by some random distribution, here's what we think could happen". What different forecasters disagree about…

I will veer this off into the dreaded political territory even though this is mostly a technical discussion. The Democratic Party proved it was not as progressive as they thought as Sanders lost the primary. The reality is, the country as a whole is also not as liberal either, regardless of what these pollsters are asking people. You think the party is youthful, and ready for progressive ideas, but alas, the party wh…

> Anyway, if you want my hot take, the conditional forecasting is to save their ass on election night from being embarrassingly wrong again.

Well Nate Silver wrote a full critically acclaimed book about why these types of forecast are more useful (and accurate) in reality because they account for uncertainty - he has been doing this for years, ever since he used to write similar algorithms to help bookies pick odds for sporting events, so I think your hot take isn’t based in any world of facts or knowledge on this.

Don’t trust a forecaster that says with certainty that a certain candidate will win, unless they have also bet their life’s earnings on it. Showing your statistical confidence level isn’t a bad thing.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#222

Post by pierrefermat1 » Mon, Oct 26, 2020, 8:26 AM UTC

yodon wrote: post #153

screye wrote: post #145

I disagree with the author on the idea that tail is too fat for isolated anomalies. There are most certainly events that can happen, which may lead to a red California or a blue Alabama. Presidential assassination, war, video proof of something incredibly heinous (pedophilia?), etc. can absolutely lead to these outcomes. You don't even have to go that far back. Nixon and Reagan flipped states like no-one's business.…

I've always liked Enrico Fermi's attitude on this. When you're Enrico Fermi, you get to say things like "One data point gives you a curve. Two data points gives you the distribution about the curve."

Curious is there a source for this? It is meant as satire?

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#223

Post by chki » Mon, Oct 26, 2020, 9:15 AM UTC

jerkstate wrote: post #214

gamblor956 wrote: post #135

Earlier quoted context omitted.

While polarization is bad now, it's nowhere close to historical extremes, and it's not even as bad as it was during the post-Vietnam era only a few decades ago. As for fear of doxing: plenty of people openly supported and voted for George Wallace (a noted white Supremacist) back in the day, and even Roy Moore (accused pedophile) just 2 years ago. Proud Boys members openly pose for the cameras even as they espouse rac…

As a data point, I live in a conservative-leaning area of a purple/blue state. Along my normal driving routes I had seen quite a few, but over the past few weeks they have mostly been taken down (in every case, other republican candidate signs still stand). Scenario one, Trump supporters supported him all the way up until now, enough to donate to his re-election campaign to buy a yard sign, and in the final weeks of…

Similar to how "not every Trump Voter is a >>Proud Boy<<" not every Biden supporter is a yard-sign stealing communist. Actually the extremists are in the minority in both groups. If you start going down that road and base your vote on how bad the worst people on the other side are, democracy is pretty much collapsing already.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#224

Post by runawaybottle » Mon, Oct 26, 2020, 10:31 AM UTC

Closi wrote: post #221

runawaybottle wrote: post #187

Earlier quoted context omitted.

I will veer this off into the dreaded political territory even though this is mostly a technical discussion. The Democratic Party proved it was not as progressive as they thought as Sanders lost the primary. The reality is, the country as a whole is also not as liberal either, regardless of what these pollsters are asking people. You think the party is youthful, and ready for progressive ideas, but alas, the party wh…

> Anyway, if you want my hot take, the conditional forecasting is to save their ass on election night from being embarrassingly wrong again. Well Nate Silver wrote a full critically acclaimed book about why these types of forecast are more useful (and accurate) in reality because they account for uncertainty - he has been doing this for years, ever since he used to write similar algorithms to help bookies pick odds f…

I think it’s certainly more grounded in reality if you realize 538 is basically finished if they miss the mark again.

If you listen to what they say, they admit they were not able to measure for the no-colllege male demographic in 2016, or in other words, they couldn’t model identity politics. Why couldn’t they do that? I’m not sure, but they are certain they can this time around because they saw the 2016 data and now believe they have more complete data to not make the same mistake again.

They are looking at elections as if there are hundreds of millions of elections that happen every day and the data speaks for itself. No sorry, there’s very few elections to extrapolate the way they are doing it, and you really need to do sociopolitical analysis of things like a demographic identity bloc (no-college whites that feel some way about things) that really get you the accurate undercurrents that can sway an election.

Lastly, it doesn’t take a genius to sit there at 10pm on election night and go ‘well if Florida and Michigan went this way, then probably so will these other states in flux’. ‘Our forecast becomes more accurate as we get the actual poll closing numbers on election night’, ah I see, you’re all geniuses, I should have known.

Anyways, we’ll know soon enough.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#225

Post by disgruntledphd2 » Mon, Oct 26, 2020, 12:25 PM UTC

swader999 wrote: post #213

StreamBright wrote: post #164

Earlier quoted context omitted.

It pretty much means nothing. These sort of models produced wrong result again and again. For me the biggest question mark if that if we know that recent (last 5) elections were very close how can you predict somebody winning with 93% chance? Maybe I do not understand something here.

Agree. Polls are a bad metric to rely on. Only 2% of those asked respond. Impossible to get a statistical sample with that. People are bad at predicting their future behavior. They are dishonest, they don't know or just don't want to tell you. There's a huge class divide right now. The bigger the class divide the worse polls are historically. And we know they are wrong this time around. None of the early voting margi…

Sure, these are all concerns. However, as long as they are not systematic errors for or against one candidate, they end up not mattering very much.

Andrew Gelman (the author of this post) has also done a bunch of work on how different parties supporters become more/less likely to respond to polls based on what the current results are, which has been incorporated into the newer forecasts.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#226

Post by noirbot » Mon, Oct 26, 2020, 12:45 PM UTC

RivieraKid wrote: post #168

noirbot wrote: post #125

Earlier quoted context omitted.

I think the question is if it matters to the predictive accuracy of the model. Just because it puts out results you can't envision actually happening on the margins doesn't mean they can't happen, or that they can't be valuable in presenting a holistic result. It's clear that the models are tuned differently, but from Silver's replies in the PS's, it seems that he's ok with these artifacts being part of the model.

Yes, it increases the state-level and national uncertainty intervals (Andrew Gelman has talked about this several times on his blog), which improves Trump's odds.

Sure, but that's not necessarily wrong. Any decision in the model will change Trump's odds in one way or another. The question is if it makes it closer to the (unknowable) real odds.

Just because intuition says it should be longer odds for Trump doesn't mean that's right.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#227

Post by wtallis » Mon, Oct 26, 2020, 12:53 PM UTC

runawaybottle wrote: post #224

Closi wrote: post #221

Earlier quoted context omitted.

> Anyway, if you want my hot take, the conditional forecasting is to save their ass on election night from being embarrassingly wrong again. Well Nate Silver wrote a full critically acclaimed book about why these types of forecast are more useful (and accurate) in reality because they account for uncertainty - he has been doing this for years, ever since he used to write similar algorithms to help bookies pick odds f…

I think it’s certainly more grounded in reality if you realize 538 is basically finished if they miss the mark again. If you listen to what they say, they admit they were not able to measure for the no-colllege male demographic in 2016, or in other words, they couldn’t model identity politics. Why couldn’t they do that? I’m not sure, but they are certain they can this time around because they saw the 2016 data and no…

> If you listen to what they say, they admit they were not able to measure for the no-colllege male demographic in 2016, or in other words, they couldn’t model identity politics. Why couldn’t they do that? I’m not sure,

You seem to have a fundamental misunderstanding of what FiveThirtyEight is trying to model, versus what pollsters are trying to model with the numbers they publish that FiveThirtyEight consumes. The kind of demographic weighting you're complaining about FiveThirtyEight being bad at is something the pollsters do, and is outside the scope of FiveThirtyEight's forecasting models.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#228

Post by rsynnott » Mon, Oct 26, 2020, 1:24 PM UTC

perryizgr8 wrote: post #211

rsynnott wrote: post #191

Earlier quoted context omitted.

> But people read “90% chance” as “definite win”. I don’t actually know what anyone should or could do about that. 538 are aware of the problem, and combating it with a cartoon fox (and better visualisations).

I don't know if this statement is serious. How does a cartoon fox make me think differently about the numbers?

I was mostly joking; the real improvement is the better visualisations. The cartoon fox is just there as a reminder.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#229

Post by rsynnott » Mon, Oct 26, 2020, 1:31 PM UTC

runawaybottle wrote: post #224

Closi wrote: post #221

Earlier quoted context omitted.

> Anyway, if you want my hot take, the conditional forecasting is to save their ass on election night from being embarrassingly wrong again. Well Nate Silver wrote a full critically acclaimed book about why these types of forecast are more useful (and accurate) in reality because they account for uncertainty - he has been doing this for years, ever since he used to write similar algorithms to help bookies pick odds f…

I think it’s certainly more grounded in reality if you realize 538 is basically finished if they miss the mark again. If you listen to what they say, they admit they were not able to measure for the no-colllege male demographic in 2016, or in other words, they couldn’t model identity politics. Why couldn’t they do that? I’m not sure, but they are certain they can this time around because they saw the 2016 data and no…

> If you listen to what they say, they admit they were not able to measure for the no-colllege male demographic in 2016, or in other words, they couldn’t model identity politics. Why couldn’t they do that? I’m not sure, but they are certain they can this time around because they saw the 2016 data and now believe they have more complete data to not make the same mistake again.

I think you possibly misunderstand what 538 _do_ a bit. Their data is based on polling, so they can only work on what the pollsters do. Historically, pollsters didn't pay that much attention to education, beyond using income or class as a proxy for it; one middle-class white man was pretty much like another. This worked quite well historically, but no longer does (and it's not just a US phenomenon; it was also a contributor to polling problems for Brexit, notably).

In their current model, 538 assume a higher rate of uncertainty than last time round; also, some pollsters now model education. But really there's not that much they can do about stuff that pollsters don't ask about.

Re: Reverse-engineering the problematic tail behavior of Fivethirtyeight forecast

#230

Post by rsynnott » Mon, Oct 26, 2020, 1:37 PM UTC

pessimizer wrote: post #199

sequoia wrote: post #175

As a US voter I am very frustrated With and tired of this obsession with election forecasting. Is it going to influence whether or not you go out and vote? If not, what is the point of it? What value does something like fivethirtyeight add to our democracy, if any? Is this motivation the same as that of diving deep into baseball stats or Star Wars starship engineering, just like “nerding out” for its own sake? Contra…

There's nothing scientific about any of this trash. It's a weird conflation of the degree of the incompetence of pollsters and the degree to which opinions can be changed within a span of time . And it's completely unfalsifiable. When Trump wins, the true believers will say "We gave him a 9.684% chance of winning! It's only your ignorance that makes you think we were wrong" and they go back to poring over their race…

> When Trump wins

So, wait, you're offended by the election modellers making a prediction, and yet you yourself are making a prediction? What's yours based on? Time machine?