Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

It seems less intellectually confused once one recognizes that it's mostly just a plug for the author's company: he links to his product at both beginning and end of the 'article.'

I'm confused by the existence of this article. Comparing the accuracy of two probabilistic predictors is pretty much a solved problem. (At least in situations where the two predictors have both made predictions about a fairly large number of events in the same set, S.) Just add up the log scores for both predictors and see which does better. [1]

[1] https://en.wikipedia.org/wiki/Scoring_rule#Logarithmic_scori...

Another thought: Although the author didn't lie about its meaning, I thought the graph "Stated Probabilities Compared with Average Portions" was visually misleading. At a casual glance, it appeared that 538 was all over the place with its senate predictions. But if you look closer, you can see that many of the red data points are at 0 or 0.5 or 1. This suggests that these data points represent the outcome of just 1 or 2 elections, which isn't really enough to say that 538 is doing particularly badly.

This "Twitter fight" began with Nate Silver talking to the media, trying to convey both his prediction and its uncertainty:

https://thehill.com/homenews/sunday-talk-shows/414759-nate-s...

> "So in the House we have Democrats with about a 4 in 5 chance of winning," Silver told ABC's "This Week." > "But no one should be surprised if they only win 19 seats and no one should be surprised if they win 51 seats," Silver added. "Those are both extremely possible, based on how accurate polls are in the real world."

Dinesh D'Souza jumps on him for hedging his bets: > And just like that, an 80 percent chance for Democratic takeover of the House goes to 50-50.

Nassim Taleb also jumps on him in the same way: > @DineshDSouza is right: you don't change a forecast from 80% to 50% under uncertainty. Second time klueless Nate Silver makes a mistake. > When someone says are event and its opposite are extremely possible I infer either 1) 50/50 or 2) the predictor is shamelessly hedging, in other words, BS. > 4- Things are actually worse: that @NateSilver538 doesn't get that if BOTH X & Non-X are "extremely possible = Probability converging to 50-50 is EXACTLY my problem w/his misunderstanding of probability in forecasting, & point of my paper. > 5- FOR THE RECORD I am not translating @DineshDSouza's point that it was 50-50: I ALSO understood ~50-50 DIRECTLY from Silver's "extremely" in the linked "The Hill".

The Twitter fight happened on Nov 4, if you want to go look.

In this article, the author notes that 538 only presents its model to readers in terms of probabilities, but the public tends to round their predictions up and down.

> This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are judged on how many events with forecasted probabilities above and below 50% happened or didn’t respectively (in a binary setting)

This isn't the public's fault, says the author, it's 538's fault for not clearly presenting the uncertainty in the prediction.

Additionally, the author writes, the 538 polling-based model swung too wildly in the 2016 election. In doing so, it failed to present a single prediction on which it could be judged. It also failed to acknowledge that unforeseen events could change the polls.

Personally, I'd argue that the swinginess of the conveyed the uncertainty of the polls. I spent the last month of 2016 citing Nate Silver to argue on Facebook and Reddit that people were overestimating the certainty of the election forecasts. Fifteen percent of the electorate was up for grabs with just two weeks left in the election! (about 7.5% undecided; and 7.5% saying they'd vote 3rd party)

> This is Taleb’s primary argument; FiveThirtyEight’s predictions do not behave like probabilities that incorporate all uncertainty and should not be passed off as them.

Nate Silver is being blamed both for trying to convey to the public that his predictions involve uncertainty and for the public not perceiving the uncertainties in his predictions.

Yes, agreed. I noticed that potential interpretation, mostly because I already knew the distinction. Now suppose you didn’t already know.

>In this article, the author notes that 538 only presents its model to readers in terms of probabilities, but the public tends to round their predictions up and down.

The only way you could get "50-50" from that is if you read "the Democrats get 51 seats" as "the Democrats win the House", because without that inference, he didn't say anything about the chance that they win the House. He compared a very lopsided win to a slight loss, and those were equally likely.