Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nassim Talebs case against Nate Silver is bad math

m.nautil.us

Post reply on HN
Print view View original

151–160 of 213 posts

Re: Nassim Talebs case against Nate Silver is bad math

#151

Post by perl4ever »

LeanderK wrote: post #98
steve19 wrote: post #66

Earlier quoted context omitted.

In many industries if we redefined probability it would be considered malpractice or fraud. We expect our doctors not to tell us we have a good chance of surviving surgery when we only have a 20% chance of surviving. I would be fraud (I assume) for a lottery to tell someone buying a ticket that they have a good chance of winning millions (0000000.1% probability that you will win but a 75% probability that it will be…

i thought more of a technical argument, if this comment is based on stuff like "Event A and not A are both extremely likely". I also think that both of your examples are not really useful, they don't violate the laws of probability. What a good chance is depends on more things than just the raw probabilities, for example i could phrase a ticket with a good chance as E(reward - price) > 0 (so its independent of its ac…

"I don't consider "Event A and not A are both extremely likely"

To me, a 20% probability is not "extremely likely", but it is "extremely possible". A 10% probability is "quite possible" or "not terribly surprising". An 80% probability is "very likely", but perhaps not "extremely likely".

Re: Nassim Talebs case against Nate Silver is bad math

#152

Post by dragonwriter »

perl4ever wrote: post #148
ham_sandwich wrote: post #49

Isn’t it just one big misunderstanding between them? Taleb thinks 538’s probabilities represent a binary option price on the event, in which case, yes, the probabilities should stay very close to 50% because the vol is so high. Whereas Silver’s models are actually saying “Based on current polls, if the election were held tomorrow, then the probability candidate X wins is Y%” and are thus allowed to swing more wildly.…

As was mentioned elsewhere in this thread (and I am aware of from reading 538) during the campaign, Silver updates separately both a probability based on the scenario of "if the election were held tomorrow" and the best prediction of the election on its actual date.

In the 2016 election season, Silver used three models, two forecast models predicting what the election would do (a “polls-only” and a “polls-plus” model incorporating non-poll data), and the “nowcast” of what would happen if the election were held at the moment of the analysis. The polls-only forecast was (at least at the end of the cycle) the headline result.

Re: Nassim Talebs case against Nate Silver is bad math

#153

Post by roenxi »

bambax wrote: post #142
tcbawo wrote: post #62

Earlier quoted context omitted.

Loads of people criticized Silver for not having "predicted" Trump's electoral victory. I imagine Silver's language was trying to hand-hold people to an understanding that even 20% probability events happen 1 out of 5 times. With so many people trying to twist and spin every headline, prediction, number, I appreciate Silver's attempt to make a reasonable guess with transparent methodologies.

I have a hard time understanding this: what does it mean to assign probabilities to one-off events?? The only way to verify probabilities predictions is to test it a great number of times. Otherwise you can say anything and never be wrong (except for 0 or 100%).

Those questions underpin big chunks of philosophy.

Assigning a probability to a one-off event is enumerating all the ways it could happen, all the ways it could not happen and assigning a probability to each of those ways. Obviously there is a lot of guesswork; but if you need to make a decision based on the future that approach gives you a much better chance of making a good decision. In practice an event will be made up of components that are more predictable than the whole, and some real uncertainties. There is a lot to be gained by thinking hard about the situation, and assigning a probability will do that.

Silly example - how do I estimate my risk of falling climbing up a set of stairs that I've never climbed before?

* Baseline risk of tripping - I have a lifetime of data.

* Increase of risk being on a staircase - I have an area I want to put my foot on (a specific step) that is about 1/3 of the area that my foot usually falls in, so that increases the risk by an amount that can be reasonably estimated.

* I will watch my foot - maybe an order of magnitude improvement in precision.

This is enough to let me estimate the risk of carrying a bulky object (that obscures my view of my feet) up a staircase. I've isolated the uncertainty (how much does visual observation change the odds) from the certainties (areas, background rate).

Now I can take that to several experts who will identify new mechanisms and tighten up my estimations on how big a deal the components are. In this way - even though the final % I come to would still be a bit arbitrary - it is starting to become a summary of what a large number of people think about the inputs to the problem and their relative magnitudes. Being able to communicate all that thinking with a single number is a miracle in its own way.

Re: Nassim Talebs case against Nate Silver is bad math

#154

Post by dmix »

iron0013 wrote: post #132
dmix wrote: post #120

Earlier quoted context omitted.

I don't know about Taleb, but Nate Silver's book on Bayesian statistics and prediction in general got me really interested in statistics from a general-reader standpoint. Where I ended up digging harder into the more math/sciencey books afterwards. The part about predicting Earthquakes and his work on sports is really good. https://www.amazon.com/Signal-Noise-Many-Predictions-Fail/dp...

I agree about Silver, and that was kind of my point! Silver knows his shit and is able to communicate it in such a way that the reader actually learns something. Taleb may or may not know anything, but all he is able to convey to the reader is the impression that they've learned something (however, he is very good at giving the reader an unearned sense of intellectual superiority).

Sure but he sounds like a decent scientist to me. While it’s true his organization is very much an entertainment vector rather than a more academic science output, which lowers the bar, that doesn’t necessarily mean the work he’s doing doesn’t have some utility and value.

Probably a lot more real value than a typical news outlet these days which pump out garbage at high speeds.

Re: Nassim Talebs case against Nate Silver is bad math

#155

Post by fineline »

oarabbus_ wrote: post #137
andrewla wrote: post #14

As I understand it, the feud in its current form started from a tweet from Dinesh D'Souza [1] in which he interpreted Nate Silver's statement that "Dems, GOP winning House are 'both extremely possible'" to mean that they had gone from an 80% chance of winning to a 50% chance. Taleb inexplicably chose this as the hill to die on [2] and wrote a lengthy and technical paper to refute, and replied (on the [2] thread) "Whe…

Taleb is simultaneously both worth reading as well as one of the most pretentious, unpleasant asses I've ever read. The Black Swan was mostly fantastic, but Antifragile was mostly regurgitation. The biggest problem with him is he thinks he's much, much smarter than he is, like a kind of inverse-Dunning-Kruger effect. It's odd because he is quite a smart man in the first place; there's no need for him to be such a pre…

I've never read Black Swan. I found Antifragile hugely thought-provoking. Knowing both, would you imagine I had anything to gain from also reading Black Swan, or do you really think they basically cover the same ground?

Re: Nassim Talebs case against Nate Silver is bad math

#156

Post by aubreyclayton »

tomp wrote: post #47
ska wrote: post #32

Earlier quoted context omitted.

He remains a real charmer I see.

But is he wrong?

He is wrong in that the sentence of his that I quoted contained two statements: "when the volatility of the underlying security increases, [1] arbitrage pressures push the corresponding binary option to trade closer to 50% and [2] become less variable over the remaining time to expiration." His calculation proves [1], but it's [2] that is the basis of his criticism of Silver. And it's just not true, as can be seen even in a simple random-walk model.

Re: Nassim Talebs case against Nate Silver is bad math

#157

Post by comex »

ikeboy wrote: post #130
galby wrote: post #126

Earlier quoted context omitted.

I think most bayesians would disagree with you on this point :) It's not incoherent to analyze the posterior distribution by reporting a credibility interval instead of a single number (for example the posterior median).

An interval to estimate a parameter is different from an interval to represent a probability. You can't have a range for a probability, you can have a range for a parameter you're trying to estimate.

Well, in theory you can divide the uncertainty into two parts, by supposing that the outcome is driven by a fundamentally stochastic process and you also have imperfect information about the parameters of that process. For example, the outcome could be determined by a coin flip but you’re not sure whether the coin is fair. In that case, there is a “true” probability of heads based on the nature of the coin, and separately, a Bayesian observer can have a probability distribution for (i.e. representing their beliefs about) the value of the true probability. The observer could then come up with a single number representing their belief in heads by taking the expected value of that probability distribution, and if they just want to gamble on the outcome, that number would be all they need. But in order to correctly update their beliefs given future information about the parameters, they have to remember the original probability distribution; they also might just be curious about the nature of the underlying stochastic process, in addition to the final outcome.

How well that models an actual election is debatable, of course, but I think it does model it to some degree. In reality, there are not two stages but multiple, and none of those stages are necessarily fundamentally stochastic; rather, you just need exponentially more information to predict one stage than to predict the previous one, and without that information you may as well treat it as stochastic. For example, if I’m about to flip a coin, a god with exact knowledge of the state of my body and brain, the air currents in the room, etc. might be able to predict how I’ll throw it and how it will fall, but mere mortals have to treat a coin flip as random. Similarly, a god with exact knowledge about the state of the universe might be able to predict an election result eons in advance… though quantum randomness might trip them up. Getting more down to earth, if you just could poll every American about their political beliefs, you could make much better predictions than you can with real polls, which have to take random samples and thus accept some level of stochastic polling error. On the other hand, polls can also suffer from methodological error, which is fundamentally different in nature; it can be highly pernicious, but does require a smaller quantity of information to correct for. And so on.

Re: Nassim Talebs case against Nate Silver is bad math

#158

Post by ohashi »

Bjartr wrote: post #53
tshaddox wrote: post #39

Earlier quoted context omitted.

Yeah, "extremely possible" is perhaps a poor choice of words, and could be interpreted uncharitably as a hedge against any outcome, but I think there's an obvious charitable interpretation too: that things that have a 20% chance of occurring turn out to occur around 20% of the time. No one would be blown away by surprise if I proclaimed "SIX!" then rolled a six-sided die and got a six, but that's even less likely tha…

Reminds me of the old essay on "Words of Estimative Probability" from the CIA https://www.cia.gov/library/center-for-the-study-of-intellig...

That's a great table they have with associated words and numerical probability associated with them.

Re: Nassim Talebs case against Nate Silver is bad math

#159

Post by ikeboy »

comex wrote: post #157
ikeboy wrote: post #130

Earlier quoted context omitted.

An interval to estimate a parameter is different from an interval to represent a probability. You can't have a range for a probability, you can have a range for a parameter you're trying to estimate.

Well, in theory you can divide the uncertainty into two parts, by supposing that the outcome is driven by a fundamentally stochastic process and you also have imperfect information about the parameters of that process. For example, the outcome could be determined by a coin flip but you’re not sure whether the coin is fair. In that case, there is a “true” probability of heads based on the nature of the coin, and separ…

Sure, but as a good Bayesian you should be willing to bet based on that expected value, not based on the spread.

Re polls: still more complicated than that, even with 100% polling you'd still have response error and non-voters. Much of the difference between polls is their assumptions about demographics of voters.

Re: Nassim Talebs case against Nate Silver is bad math

#160

Post by mr_toad »

bambax wrote: post #142
tcbawo wrote: post #62

Earlier quoted context omitted.

Loads of people criticized Silver for not having "predicted" Trump's electoral victory. I imagine Silver's language was trying to hand-hold people to an understanding that even 20% probability events happen 1 out of 5 times. With so many people trying to twist and spin every headline, prediction, number, I appreciate Silver's attempt to make a reasonable guess with transparent methodologies.

I have a hard time understanding this: what does it mean to assign probabilities to one-off events?? The only way to verify probabilities predictions is to test it a great number of times. Otherwise you can say anything and never be wrong (except for 0 or 100%).

It’s not that the election itself is subject to random uncertainty so much as the data that they are using to make the prediction is.

The data might favour one candidate, but even assuming it is unbiased and representative, it is only a random sample, and there is a chance it could be randomly wrong.

Post reply on HN