Live data from Hacker News

Nassim Talebs case against Nate Silver is bad math

m.nautil.us

191–200 of 213 posts

Re: Nassim Talebs case against Nate Silver is bad math

#191
post #62

Earlier quoted context omitted.

Yeah, "extremely possible" is perhaps a poor choice of words, and could be interpreted uncharitably as a hedge against any outcome, but I think there's an obvious charitable interpretation too: that things that have a 20% chance of occurring turn out to occur around 20% of the time. No one would be blown away by surprise if I proclaimed "SIX!" then rolled a six-sided die and got a six, but that's even less likely tha…

Loads of people criticized Silver for not having "predicted" Trump's electoral victory. I imagine Silver's language was trying to hand-hold people to an understanding that even 20% probability events happen 1 out of 5 times. With so many people trying to twist and spin every headline, prediction, number, I appreciate Silver's attempt to make a reasonable guess with transparent methodologies.

I remember him saying before the election that the odds of Trump winning we're roughly the odds of a randomly picked day being a Saturday. Which always seemed pretty clear to me - Saturdays definitely happen, Trump could definitely happen. And then it did, and it was nothing like a Saturday, not at all.

Re: Nassim Talebs case against Nate Silver is bad math

#192
post #142
post #62

Earlier quoted context omitted.

Loads of people criticized Silver for not having "predicted" Trump's electoral victory. I imagine Silver's language was trying to hand-hold people to an understanding that even 20% probability events happen 1 out of 5 times. With so many people trying to twist and spin every headline, prediction, number, I appreciate Silver's attempt to make a reasonable guess with transparent methodologies.

I have a hard time understanding this: what does it mean to assign probabilities to one-off events?? The only way to verify probabilities predictions is to test it a great number of times. Otherwise you can say anything and never be wrong (except for 0 or 100%).

You can bet money on one off events. People and bookies do this frequently and you can see if they loose or win.

But the betting odds depend on how much knowledge you have - for someone with perfect knowledge the odds may be 0 or 1, if you know nothing you might guess 50:50 for practical purposes.

Re: Nassim Talebs case against Nate Silver is bad math

#193
post #142
post #62

Earlier quoted context omitted.

Loads of people criticized Silver for not having "predicted" Trump's electoral victory. I imagine Silver's language was trying to hand-hold people to an understanding that even 20% probability events happen 1 out of 5 times. With so many people trying to twist and spin every headline, prediction, number, I appreciate Silver's attempt to make a reasonable guess with transparent methodologies.

I have a hard time understanding this: what does it mean to assign probabilities to one-off events?? The only way to verify probabilities predictions is to test it a great number of times. Otherwise you can say anything and never be wrong (except for 0 or 100%).

You can't verify probabilities for any single given event, but you can look at the outcomes for all events for which predicted probabilities were given, and use that to estimate the accuracy of the methodology/models used.

So, for example, when 538 said that Trump had a 30% chance of winning, its hard to say how accurate that was given it was a one time event. But if you look over all of their election predictions, and look at those they predicted to win 30% of the time, about 30% of them should have been victories (with some margin for error, of course). If 50% were victories, or only 10%, then maybe that would indicate their models aren't doing so hot.

Re: Nassim Talebs case against Nate Silver is bad math

#194

Earlier quoted context omitted.

He is wrong in that the sentence of his that I quoted contained two statements: "when the volatility of the underlying security increases, [1] arbitrage pressures push the corresponding binary option to trade closer to 50% and [2] become less variable over the remaining time to expiration." His calculation proves [1], but it's [2] that is the basis of his criticism of Silver. And it's just not true, as can be seen ev…

"[2] become less variable over the remaining time to expiration ... And it's just not true, as can be seen even in a simple random-walk model." Dear @aubreyclayton, I'm genuinely interested in seeing how you arrived at this conclusion. Would you kindly share the proof, or at least explain the logic behind it. Thank you.

Sure. If you've studied stochastic processes like Brownian motion, check out the derivation starting with the last paragraph on p.4 of this note: https://drive.google.com/file/d/1tQj4ZGja6jKADJGiFj-dcQAirWo...

If that's foreign to you, you might be interested in this write-up I did about election-forecasting in which I considered the same example, just in discrete time rather than continuous time: http://nautil.us/issue/70/variables/how-to-improve-political...

The basic idea is that if you increase the volatility of a random-walk process, say by making the step-sizes larger, that won't actually make the probability of finishing above where you started any less (or more) volatile. The higher volatility means that from any given starting point your final resting place is more dispersed, but you're also more likely to range farther from home as you go. The two effects exactly cancel. Taleb's critique misses the second part of that.

Re: Nassim Talebs case against Nate Silver is bad math

#195

So... I wrote this. Maybe I can clarify a little what I meant by the statement "Since Silver’s forecasts begin with probability models, it’s safe to assume they obey all the rules, including Bayes’, and would be arbitrage-free.", since this seems to be confusing some people: What I'm addressing here is Taleb's claim that Silver's probabilities would allow for arbitrage (i.e., riskless profit, not just profit on avera…

Needing help with one part of this. Say in the simplest case of a coin flip and two instants in time I assess probabilities: t=0: P(heads) = 0.75, P(tails) = 0.25 t=1: P(heads) = 0.60, P(tails) = 0.40 These are consistent (or behave as probabilities or what have you) at each time (separately), so de Finetti's argument covers each (separately). Does this alone somehow protect from arbitrage over time? Or is the martin…

So, if I knew ahead of time what your price was going to be tomorrow, I would have a clear arbitrage strategy. But that's not a fair example. The question is: what do I know at t=0 about what your price could be at t=1?

A better example is this: suppose you're creating a probabilistic forecast of the chance of a coin coming up Heads twice. Suppose this represents a win. You say, "according to my model, P[H1 and H2] = 0.3." And suppose you also say P[H1] = 0.4

Then I ask you: imagining the first flip comes up Heads, what will your price for the second Heads be then? The only arbitrage-free price you can quote me is your conditional probability: P[H2 | H1] = P[H1 and H2]/P[H1] = 0.75. Anything else allows me to buy and sell bets and make a riskless profit.

Now, note that if H1 occurs, P[H2 | H1] is now the updated chance of getting H1 and H2. You observed some information and updated your probability accordingly. Also note that according to your model, the chance of H1 happening was 0.4, so the possible prices of "H1 and H2" after one coin flip were 0.75 with probability 0.4 and 0 with probability 0.6 (if the first flip were tails, "H1 and H2" is impossible). So the expected price of "H1 and H2" after one flip is (.75)(.4) + (0)(.6) = 0.3, which is also your price at t=0. That means the probability/price is a martingale according to your probability assignments, which, according to the Fundamental Theorem of Asset Pricing, again means no arbitrage is possible, unless we disagree on what events have probability zero.

So, the election forecasts are like this except we don't get to see all the inner workings. Nate doesn't quote things like the probability of the polls moving by a certain amount each day. We just get to see things like P[H1 and H2] before and after the first coin flip. But as long as it's possible that those come from a consistent set of conditional probabilities and Bayesian updating, there's no chance for arbitrage.

Re: Nassim Talebs case against Nate Silver is bad math

#196

Earlier quoted context omitted.

You basically have it right. But it's not a misunderstanding. Taleb understands Silver's approach, but thinks it's BS, entertainment not forecasting. He's right, but I don't know why he cares. The problem isn't that Silver doesn't know math, it's that his goal is to entertain, not predict.

Taleb calling someone else an entertainer is a bit rich, sort of like a clown telling someone to have dignity. I have no opinion on Silver whatsoever, but Taleb is a bloviating ass who has gone from popular milquetoast observations, to climbing fully up his own backpassage. That this all began with the likes of Dinesh D’Souza doesn’t help. I realize that Black Swan is popular here, but it was horrendous. A single ess…

You probably haven't even heard of his Technical Incerto, which is hundreds of pages of underlying principles and explanations. Literally what you are 'asking' for. Maybe keep your strong opinions to yourself before you embarass even further

Re: Nassim Talebs case against Nate Silver is bad math

#198
post #190
post #149

Earlier quoted context omitted.

You don't need the price to revert, you just need to have a better estimate of the underlying probability.

Oh so the coin-flipper is not setting the price, but trying to predict it? I think at least I understand what you're explaining. I don't see how that is a useful model for what is going on though. Predicting the result of a vote for a day when it's not going to be held is a form of entertainment that can't be verified. Is Silver claiming any more than that?

Yes, Silver is absolutely trying to forecast elections months in advance. Taleb's argument is that the polls at that point are largely noise, and so Silver is updating too much on them when he should basically ignore most of them until shortly before the election.

Re: Nassim Talebs case against Nate Silver is bad math

#199
post #139

As I understand it, the feud in its current form started from a tweet from Dinesh D'Souza [1] in which he interpreted Nate Silver's statement that "Dems, GOP winning House are 'both extremely possible'" to mean that they had gone from an 80% chance of winning to a 50% chance. Taleb inexplicably chose this as the hill to die on [2] and wrote a lengthy and technical paper to refute, and replied (on the [2] thread) "Whe…

> But in this case I'm having trouble following what his criticism even is A piece by Isaac Faber [1] from December last year clearly described what Taleb's position is. A summary paragraph towards the end: > Here we can say, with some confidence that FiveThirtyEight predictions are not always reliable probabilities. However, they masquerade as one, being between 0 and 1 and all. This is Taleb’s primary argument; Fiv…

I confess, I still don't follow the critique. Let's take the output of a skin-in-the-game model, like a binary futures market. This also predicts numbers between 0 and 1, that look a lot like probabilities, but by the argument presented here, they are not probabilities, and if you look at the output of a futures market, then you are being scammed because the market itself is not giving a decision boundary; it's just outputting the last trade price.

Yes, this allows a forecaster to say that even if the Is the criticism just that we shouldn't look at Silver as a person with anything interesting to say because he won't commit 100% to one side of an argument? I honestly don't understand what the critique is, and this link I don't find helpful.

Re: Nassim Talebs case against Nate Silver is bad math

#200
post #27

Earlier quoted context omitted.

The best criticism I can come up with is that 538 probably doesn't do a sufficient job of explaining to its audience that their published probabilities are predictions for what would happen if the event occurred today . This seems to be one of Taleb's (and others') criticisms of their predictions: that they don't consider the probability of major unexpected events occurring between now and the event (like an athlete…

> The best criticism I can come up with is that 538 probably doesn't do a sufficient job of explaining to its audience that their published probabilities are predictions for what would happen if the event occurred today I don't think that's true. While 538 does such predictions (the “nowcast”), they also predict actual results (the “polls-plus forecast” and the “polls-only forecast”), the default view is a forecast (…

This is not correct, or at least it isn't any longer. The current predictions by 538 are the "lite" (polls only), the "classic" (polls + other data driven indicators) and the "deluxe" (polls + other data driven indicators + expert opinion). All of these are predictions based on "what would be the outcome of the election be were it to be held today". They just incorporate different data and complexity in their models.

Previously, the reported predictions were the "nowcast", "polls-only", and "polls-plus". But despite their names, I believe those too, whatever 538 may have described them as, were "predictions for outcomes as if the event occurred today". In fact, were they not the case, then Taleb's argument would actually have merit, despite his obtuse attempts to explain it.

Essentially, his argument is that, if you are predicting a particular outcome to an event (say, Clinton winning the election) a year from now with a certainty of 90%, and 3 months from now, your prediction is 75%, and 3 months after that is 85%, and then right before the election, is 70%, is evidence of a flawed model (keep in mind, I'm making these numbers up).

If your prediction is 90% a year out, it should fluctuate much as time goes on, because 90% is very strong certainty. One may counter that as time goes on, public opinion may change, there are changes in news coverage, and external events like the economy or scandal can make a big impact, and all of that is absolutely true. But the point is that, if in light of new information coming forward, your model predictions repeatedly exhibit volatility, than your model failed to adequately account for the uncertainty as a result of future events, and you were never actually 90% sure in the first place. By necessity, a model predicting a distant event in which lots of unknown factors can cause it to fluctuate before it occurs will either be fairly stable as time goes on (meaning you indeed were correct that there was a high chance of a Clinton win), or should give much more hedged estimates of a win (meaning your probabilities a year out would be much closer to 50-50).

Taleb's point is that 538's models are too volatile to actually be accurate when they say there is an X% chance someone wins an election a year, or 6 months from now. However, his argument does not apply in the case where the prediction is that there is an X% chance someone wins an election, were the election held today. Instead he seems to dismiss that that type of prediction is worthless, which is simply wrong and foolish in my opinion. There is lots to be learned about the current political climate from those predictions.

So its entirely possible both parties are partially wrong and right. Taleb is correct that 538's models are too volatile to be accurate predictions far out from the election, but it assumes that the model is a prediction of what will happen on election day, rather than what would happen today. He also is throwing out the baby with the bathwater by insisting that that is the only kind of political prediction that has any value. On the other hand, 538 has not been very clear on if their forecasts are predictions of what happens on election day or what happens were election day today. If its the former, then their models have issues and Taleb has at least some validity to his argument. If its the latter, then Taleb's argument is no longer applicable.

Post reply on HN