Live data from Hacker News

Nassim Talebs case against Nate Silver is bad math

m.nautil.us

171–180 of 213 posts

Re: Nassim Talebs case against Nate Silver is bad math

#171

Earlier quoted context omitted.

Maybe this was true of past elections, but their 2018 house forecast had a “how this forecast has changed” section that showed their historical predictions throughout election season. It’s still visible here: https://projects.fivethirtyeight.com/2018-midterm-election-f...

> Maybe this was true of past elections, but their 2018 house forecast had a “how this forecast has changed” section that showed their historical predictions throughout election season. It has not changed. They did this the whole way back when it was an independent blog before the NY Times (later ESPN then ABC News) picked them up.

Where are they? Five different posters have claimed this, but zero have been able to link to the historical predictions.

Re: Nassim Talebs case against Nate Silver is bad math

#172

Earlier quoted context omitted.

> It is worth noting that 538 doesn't keep a historical record of their predictions, making it difficult to really see how they predict outcomes coming up to a conclusion. You can only see the latest/final prediction. It's worth noting that you are completely wrong: not only do they keep such a record, they also provide a handy chart of it on the latest prediction page so you can easily see exactly how the prediction…

Where? Go ahead and link to it.

Hey I don't have a dog in this fight, but I think this is the link you're looking for.

https://github.com/fivethirtyeight/checking-our-work-data

Re: Nassim Talebs case against Nate Silver is bad math

#173

Earlier quoted context omitted.

> Maybe this was true of past elections, but their 2018 house forecast had a “how this forecast has changed” section that showed their historical predictions throughout election season. It has not changed. They did this the whole way back when it was an independent blog before the NY Times (later ESPN then ABC News) picked them up.

Where are they? Five different posters have claimed this, but zero have been able to link to the historical predictions.

> Where are they? Five different posters have claimed this, but zero have been able to link to the historical predictions.

They are literally charted on the page with the current prediction, e.g., for the 2016 Presidential race (which is locked on the last forecast):

https://projects.fivethirtyeight.com/2016-election-forecast/ , under “How the forecast has changed”, it charts the forecast (by default, the win probability) for the entire time the race was being forecast.

Re: Nassim Talebs case against Nate Silver is bad math

#174

Isn’t it just one big misunderstanding between them? Taleb thinks 538’s probabilities represent a binary option price on the event, in which case, yes, the probabilities should stay very close to 50% because the vol is so high. Whereas Silver’s models are actually saying “Based on current polls, if the election were held tomorrow, then the probability candidate X wins is Y%” and are thus allowed to swing more wildly.…

You basically have it right. But it's not a misunderstanding. Taleb understands Silver's approach, but thinks it's BS, entertainment not forecasting. He's right, but I don't know why he cares. The problem isn't that Silver doesn't know math, it's that his goal is to entertain, not predict.

> You basically have it right.

GP doesn't even have Silver’s model right (he describes the “nowcast”, which was never the headline forecast model), so if he is right that that is what Taleb is referring to, that would be a pretty damning indictment of Taleb’s critique.

Re: Nassim Talebs case against Nate Silver is bad math

#175

So... I wrote this. Maybe I can clarify a little what I meant by the statement "Since Silver’s forecasts begin with probability models, it’s safe to assume they obey all the rules, including Bayes’, and would be arbitrage-free.", since this seems to be confusing some people: What I'm addressing here is Taleb's claim that Silver's probabilities would allow for arbitrage (i.e., riskless profit, not just profit on avera…

Aside (but not really): guessing this is you? https://www.youtube.com/watch?v=P6P1rjJuD_M A totally awesome set of lectures... highly recommended.

Re: Nassim Talebs case against Nate Silver is bad math

#176
post #102

Earlier quoted context omitted.

The 2016 election predictions are still posted. https://projects.fivethirtyeight.com/2016-election-forecast/...

Yes the newest one. Not the historical ones.

Maybe I misunderstand your argument, but doesn't that linked page have a section "How the forecast has changed"[1]?

[1] https://projects.fivethirtyeight.com/2016-election-forecast/...

Re: Nassim Talebs case against Nate Silver is bad math

#177
post #172

Earlier quoted context omitted.

Where? Go ahead and link to it.

Hey I don't have a dog in this fight, but I think this is the link you're looking for. https://github.com/fivethirtyeight/checking-our-work-data

Nope. I'm looking for 538's predictions. You cannot go from the raw data they used back to how they interpreted it.

Re: Nassim Talebs case against Nate Silver is bad math

#178
post #47

Earlier quoted context omitted.

But is he wrong?

He is wrong in that the sentence of his that I quoted contained two statements: "when the volatility of the underlying security increases, [1] arbitrage pressures push the corresponding binary option to trade closer to 50% and [2] become less variable over the remaining time to expiration." His calculation proves [1], but it's [2] that is the basis of his criticism of Silver. And it's just not true, as can be seen ev…

"[2] become less variable over the remaining time to expiration ... And it's just not true, as can be seen even in a simple random-walk model."

Dear @aubreyclayton, I'm genuinely interested in seeing how you arrived at this conclusion. Would you kindly share the proof, or at least explain the logic behind it. Thank you.

Re: Nassim Talebs case against Nate Silver is bad math

#179
post #142
post #62

Earlier quoted context omitted.

Loads of people criticized Silver for not having "predicted" Trump's electoral victory. I imagine Silver's language was trying to hand-hold people to an understanding that even 20% probability events happen 1 out of 5 times. With so many people trying to twist and spin every headline, prediction, number, I appreciate Silver's attempt to make a reasonable guess with transparent methodologies.

I have a hard time understanding this: what does it mean to assign probabilities to one-off events?? The only way to verify probabilities predictions is to test it a great number of times. Otherwise you can say anything and never be wrong (except for 0 or 100%).

Every event is a one-off event. In particular the Trump election was the most one-off event in recent memory :)

Kidding aside, your observation is correct, you need to perform repeated predictions. 538 does just that. They keep predicting a whole lot of outcomes, so you can check their track record. They even have a challenge for the audience, where you can record your own predictions and compare them with their corresponding predictions (see for example [1] for NFL games). The scoring of the predictions uses the Brier score [2], which is just a version of R-squared for classification problems.

In the case of the November 2016 election, other prediction sites were giving the Trump team a 1-2% chance, 538 was giving them a 20% change. Considering that, 538 comes out as the clear winner.

Which brings us to what I consider to be the best measure of prediction accuracy (my own invention, it doesn't have a scientific name yet): you can compare two sequences of predictions (let's say yours agains 538) by simulating bets at some mid-odds level. For example if I say Trump's probability of winning 2020 is 45% and you say it's 35%, then I'm willing to give you 2-to-3 odds, while you're willing to take that. We then see who stays ahead most of the time in this betting simulation.

For what is worth, here's a recent survey of different scoring types for predictions: "Assessing the performance of prediction models: a framework for traditional and novel measures" [3]

[1] https://projects.fivethirtyeight.com/nfl-predictions-game/ [2] https://en.wikipedia.org/wiki/Brier_score [3] https://www.ncbi.nlm.nih.gov/pubmed/20010215

Re: Nassim Talebs case against Nate Silver is bad math

#180

So... I wrote this. Maybe I can clarify a little what I meant by the statement "Since Silver’s forecasts begin with probability models, it’s safe to assume they obey all the rules, including Bayes’, and would be arbitrage-free.", since this seems to be confusing some people: What I'm addressing here is Taleb's claim that Silver's probabilities would allow for arbitrage (i.e., riskless profit, not just profit on avera…

Aside (but not really): guessing this is you? https://www.youtube.com/watch?v=P6P1rjJuD_M A totally awesome set of lectures... highly recommended.

Yep! Thanks!
Post reply on HN