Why we should care about the Nate Silver vs. Nassim Taleb Twitter war • Hacker News Board

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#141

Post by huffmsa » Tue, Dec 18, 2018, 2:12 PM UTC

ubernostrum wrote: post #121

zimablue wrote: post #87

To try and explain Taleb's argument as simply as possible: If someone tells you today that an event is 100% certain, and tomorrow tells you that it is impossible, then you intuitively will not trust their forecasting. If it's certain, then it's certain not to be impossible tomorrow Now if someone tells you something is a 90% chance today, and a 10% chance tomorrow, again you don't trust their predictions. There is a…

I've always treated 538's forecasts as meaning nothing more or less than "We have a model, we fed the current data into it and ran N simulations of the event, and in X% of those simulations, this was the result". Which, as I understand it, is literally how they're generated. Any discussion of the accuracy of the model (which should be the only thing that admits debate) has to be retrospective. Just saying "well the f…

Yes but that's not how the proles understand the blog. They take it as "x has a y % chance of happening".

Which is Taleb's beef with it. The commoner thinks its Silver putting his money on something, but when it doesn't happen he says "well I didn't say _that_".

And it in the day and age of more robust machine learning models, that the 538 models are so volatile is a pretty weak excuse.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#142

Post by nscalf » Tue, Dec 18, 2018, 2:15 PM UTC

dagw wrote: post #75

azernik wrote: post #35

Earlier quoted context omitted.

That is actually an issue they've acknowledged and tried to address. People also tended to see percentages and think they reflected vote percents. To address this, for the 2018 election cycle 538 started displaying their odds as numerical ratios (5 in 9, for example) to try to reduce the aura of certainty.

538 started displaying their odds as numerical ratios Did that actually make things clearer to people? Speaking only for myself, I find ratios quite hard to works with and reason about. I certainly couldn't tell you instantly for example if 5/9 is more or less than 4/7 or how big the difference between the two are without first doing some mental arithmetic. Perhaps people in the US are more used to working with ratio…

That's the point though. It's meant to require a bit of thinking to digest the information. They didn't set out to make the information clearer and easier to understand, they set out to make the false positive of reading a percentage and considering it final results less frequent.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#143

Post by paultopia » Tue, Dec 18, 2018, 2:27 PM UTC

elgenie wrote: post #98

VintageCool wrote: post #55

Earlier quoted context omitted.

This "Twitter fight" began with Nate Silver talking to the media, trying to convey both his prediction and its uncertainty: https://thehill.com/homenews/sunday-talk-shows/414759-nate-s... > "So in the House we have Democrats with about a 4 in 5 chance of winning," Silver told ABC's "This Week." > "But no one should be surprised if they only win 19 seats and no one should be surprised if they win 51 seats," Silver add…

The final 538 pre-election forecast is available here: https://projects.fivethirtyeight.com/2018-midterm-election-f... (it moved slightly towards Dems in the last few days) Silver's point is that their model results in a probability distribution of outcomes and, at the time he was commenting, in 10% of the cases Dems won 19 or fewer seats, resulting in a GOP majority, and in 10% of cases Dems won 51 or more seats, re…

Yeah, this is the key point. I was shocked when I saw the linked twitter posts from that exchange (having not followed it at all).

Just based on that snapshot my take is even more grim. Taleb surely knows how probability distributions work. The fact that he sneeringly dismissed basic reasoning like that in the political context he did suggests not cluelessness but malice.

One thing we do know is that Taleb is very smart and very good at making money. And right now there's a lot of money to be spent pandering to the dumb nihilistic and pseudo-intellectual wing of the political right; indeed, the same people that D'Souza has been pandering to for decades.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#144

Post by JabavuAdams » Tue, Dec 18, 2018, 2:52 PM UTC

acidburnNSA wrote: post #39

mturmon wrote: post #25

This is an intellectually confused post. It is so unclear and muddled that only a charitable reader would suppose the author understands the mathematical and modeling issues here. I’m surprised the author published it in this form - you don’t want to jump into a controversy like this with such an unclear discussion. As just one example, the whole digression on a “decision boundary” is conceptually mistaken. Once you…

Peripherally, I like using nuclear reactor design analogies for Aleatory vs. Epistemic. Aleatory is the inherent randomness of a system. You can characterize it but it's often hard to reduce. For example, how much boron impurities are mixed into your steel at the time of manufacture. Epistemic is things that are knowable but you don't know with much certainty (because it's hard to measure), like the probability that…

Wow, that's so much more clear and intuitive than using the six-sided die analogy. Thanks!

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#145

Post by matt4077 » Tue, Dec 18, 2018, 2:53 PM UTC

safgasCVS wrote: post #94

mturmon wrote: post #25

This is an intellectually confused post. It is so unclear and muddled that only a charitable reader would suppose the author understands the mathematical and modeling issues here. I’m surprised the author published it in this form - you don’t want to jump into a controversy like this with such an unclear discussion. As just one example, the whole digression on a “decision boundary” is conceptually mistaken. Once you…

Unlikely. Taleb seems to like how his argument has been summarised. The crux of the issue taleb has with silver is showing probabilities without making a decision/declaration is cowardly. When asked about making a prediction he says one thing then follows it up with a “ but don’t be surprised if it could be completely random.” And that’s the point - the fact he covers his ass is the problem. Election forecasting is h…

538 had specific probability estimates for hundreds of races both 2016 and 2018, and it’s perfectly possible to evaluate them.

In fact, they regularly do so themselves: https://fivethirtyeight.com/features/how-fivethirtyeights-20...

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#146

Post by danso » Tue, Dec 18, 2018, 2:54 PM UTC

lftl wrote: post #133

OscarCunningham wrote: post #96

Earlier quoted context omitted.

I disagree. Probabilities can be useful even if they're not 0% or 100%. For example weather forecasts often give the probability of rain. My plans will be different if the probability is 20% vs 80%. My plans don't tend to vary based on who wins the election, but I imagine that that information is very useful to some people. And if your data is telling you to be 80% sure, I don't think you should fudge it and claim 10…

With the weather man, I could keep track of when it rains and doesn't. If it rains significantly more or less than 80% of the time they say there's an 80% chance of rain, I can say that their model is bad. I can't think of a similar way to evaluate Silver's election forecasting model. They very clearly aren't independent probabilities, and his model changes significantly from cycle to cycle. Was his model good in 201…

You can’t come up with a way to compare his predictions with how the elections end up turning out? Can’t you look up the results of each election?

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#147

Post by JumpCrisscross » Tue, Dec 18, 2018, 2:55 PM UTC

lftl wrote: post #133

OscarCunningham wrote: post #96

Earlier quoted context omitted.

I disagree. Probabilities can be useful even if they're not 0% or 100%. For example weather forecasts often give the probability of rain. My plans will be different if the probability is 20% vs 80%. My plans don't tend to vary based on who wins the election, but I imagine that that information is very useful to some people. And if your data is telling you to be 80% sure, I don't think you should fudge it and claim 10…

With the weather man, I could keep track of when it rains and doesn't. If it rains significantly more or less than 80% of the time they say there's an 80% chance of rain, I can say that their model is bad. I can't think of a similar way to evaluate Silver's election forecasting model. They very clearly aren't independent probabilities, and his model changes significantly from cycle to cycle. Was his model good in 201…

> I can't think of a similar way to evaluate Silver's election forecasting mode

You bucket every prediction, look at the outcome, and then confirm whether the favoured outcomes in the 8th decile actually occurred 70% to 80% of the time.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#148

Post by JabavuAdams » Tue, Dec 18, 2018, 2:56 PM UTC

zimablue wrote: post #87

To try and explain Taleb's argument as simply as possible: If someone tells you today that an event is 100% certain, and tomorrow tells you that it is impossible, then you intuitively will not trust their forecasting. If it's certain, then it's certain not to be impossible tomorrow Now if someone tells you something is a 90% chance today, and a 10% chance tomorrow, again you don't trust their predictions. There is a…

Thanks for this! Hadn't thought of this before.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#149

Post by hn_throwaway_99 » Tue, Dec 18, 2018, 2:58 PM UTC

Kurtz79 wrote: post #63

hn_throwaway_99 wrote: post #37

Earlier quoted context omitted.

Agreed. The more I read the more I got embarrassed for the author, mainly because his points were so ridiculously flimsy. I thought this was one of the worst: > Because FiveThirtyEight only predicts probabilities, they do not ever take an absolute stand on an outcome: No ‘skin in the game’ as Taleb would say. This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are j…

"Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability." Unfortunately that is the more common case, and I agree on the author on that that most people take a binary stance on polls. It should not be the case for regular readers of a website whose main topic is statistical analysis, although I imagine that just before a major election they would see a…

Fine, but Silver goes to extreme ends in my opinion to explain what a probability means. During the 2016 presidential election he was emphasizing "Hey, it's less likely than the alternative, but there is a real possibility Trump will win. There is a very plausible path for him to wrap up the electoral votes." And afterwards, I think the results showed how accurate Silver was - he gave a much higher probability to Trump winning than anyone else IIRC, and the outcome was pretty much that: An unlikely probability (many of the deciding states were very much within any possible margin of error) that actually happened.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#150

Post by matt4077 » Tue, Dec 18, 2018, 3:00 PM UTC

lftl wrote: post #133

OscarCunningham wrote: post #96

Earlier quoted context omitted.

I disagree. Probabilities can be useful even if they're not 0% or 100%. For example weather forecasts often give the probability of rain. My plans will be different if the probability is 20% vs 80%. My plans don't tend to vary based on who wins the election, but I imagine that that information is very useful to some people. And if your data is telling you to be 80% sure, I don't think you should fudge it and claim 10…

With the weather man, I could keep track of when it rains and doesn't. If it rains significantly more or less than 80% of the time they say there's an 80% chance of rain, I can say that their model is bad. I can't think of a similar way to evaluate Silver's election forecasting model. They very clearly aren't independent probabilities, and his model changes significantly from cycle to cycle. Was his model good in 201…

They do not only predict top-line presidential results, but every race, in every state, for president, House, and Senate. Non-independence is accounted for in the model, so they have no qualms about you judging them by the calibration of their predictions, I. e. You want roughly 60% of their “60%” forecasts to be right, and 40% to be wrong. If all of the races they predict 60/40 go to the more likely candidate, they themselves consider this a failure: https://fivethirtyeight.com/features/how-fivethirtyeights-20...