Why we should care about the Nate Silver vs. Nassim Taleb Twitter war • Hacker News Board

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#61

Post by waterhouse » Tue, Dec 18, 2018, 7:54 AM UTC

potatofarmer45 wrote: post #27

Nate Silver built an aleatory model whereby low probability high effect events (Comey reopening the investigation on eve of the election) are discounted because if you add enough of these potential disruptive events the model will be 50%-50% and that has no value to anybody. Who wants to read an article that said "election forecast 50-50" every day. If a 538 model says one outcome is 75% probable, it really should co…

> What is a really interesting takeaway is that if in an event where the outcome you desire based appears unlikely, your job is introduce as much previously undefined or discounted uncertainty. Ideally you engineer a black swan event or at least do what you can to make it happen. This would be fun to model from a game-theory approach instead.

In chess, if you are behind, John Nunn's two recommended strategies are "grim defence" (if your opponent's advantage is not so large as to make it easy for them to force a win) and "create confusion": create complicated tactical situations and hope your opponent makes a mistake. The farther behind you are, the more appealing "create confusion" gets by comparison.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#62

Post by paganel » Tue, Dec 18, 2018, 8:01 AM UTC

VintageCool wrote: post #58

kingkawn wrote: post #26

Earlier quoted context omitted.

Seems like in addition to whether they're more swingy than they should be, are they being presented in a way that is interpreted as more certain or authoritative than they can possibly claim?

The 2016 election was swingy and it was uncertain. In 2008 and 2012, the political media was myopically focused on the twists and turns of the horse race and called it 50-50 dead even. Nate Silver's 538 modeled the election as basically stable with only small polling shifts, and the only change in the last month was a steady decline in the remaining time that McCain (or Romney) had to significantly shift the polls. I…

>The 2016 election was swingy and it was uncertain.

I don't remember it that way, in fact I remember seeing a lot of very shocked faces in the Democratic camp when the results were beginning to take shape. I don't remember having seen similar confused reactions after any previous US presidential elections, not even after the Bush vs Gore one which was a lot closer in terms of electoral votes.

Back to the article, I think Nate Silver's failure only shows to the general public that electoral predictions are rubbish. Maybe "failure" is a strong word because he genuinely seems to be the best at what he's doing, it's just that the domain in which he's involved is turning out to be bogus. I'm sure that there was a crystal-ball viewer that was the best at what he/she was doing, it's just that crystal-ball viewing turned out to be bogus.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#63

Post by Kurtz79 » Tue, Dec 18, 2018, 8:05 AM UTC

hn_throwaway_99 wrote: post #37

mturmon wrote: post #25

This is an intellectually confused post. It is so unclear and muddled that only a charitable reader would suppose the author understands the mathematical and modeling issues here. I’m surprised the author published it in this form - you don’t want to jump into a controversy like this with such an unclear discussion. As just one example, the whole digression on a “decision boundary” is conceptually mistaken. Once you…

Agreed. The more I read the more I got embarrassed for the author, mainly because his points were so ridiculously flimsy. I thought this was one of the worst: > Because FiveThirtyEight only predicts probabilities, they do not ever take an absolute stand on an outcome: No ‘skin in the game’ as Taleb would say. This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are j…

"Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability."

Unfortunately that is the more common case, and I agree on the author on that that most people take a binary stance on polls.

It should not be the case for regular readers of a website whose main topic is statistical analysis, although I imagine that just before a major election they would see a large number of non-regulars just to check the polls.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#64

Post by darawk » Tue, Dec 18, 2018, 8:06 AM UTC

comex wrote: post #41

rsj_hn wrote: post #6

Earlier quoted context omitted.

Taleb actually makes a great point here and published a pretty cool paper about it. The point is that if a probability of the event changes too much, you can arbitrage it. E.g. assume these are payoff odds and you can sell your position before the event. He then found some nice no-arbitrage conditions that "real" probabilities must meet and showed that Nate's predictive timeline allowed arbitrage. Unfortunately Nate…

I don’t have the background knowledge to properly understand the math in that paper. But I do know that you can’t simply set bounds a priori on how much a probability can change after additional information has been gathered; no matter how pathological the swings, you can design a system, and a series of observations of that system, that would make all the forecasts (i.e. conditional probabilities) correct. Thus the…

I believe he's making a slightly more subtle point. It's not just that the forecasts are swinging too much, it's that they're swinging too much too early.

Consider an option on a stock (which is the analogy Taleb is making here). If you buy a 1 year call option on AAPL and tomorrow they announce that they beat earnings by 10%, that's not a huge deal for you. If on the other hand, you owned a 1-week expiration call, it is a big deal for you. That is, your prediction should be less sensitive to changes in environment the further out it is.

I believe he is somehow formalizing this statement, and then showing that Nate Silver's forecasts violate it, but I too don't fully understand his formalism.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#65

Post by Animats » Tue, Dec 18, 2018, 8:07 AM UTC

pmart123 wrote: post #33

I liked the first chapter of "Fooled by Randomness," but I stopped reading the book when each chapter regurgitated the same points, each more nihilistic than the last. Perhaps Taleb should concentrate on the accuracy of his own predictions: https://www.businessinsider.com/taleb-every-single-human-bei... >>> "every single human being" should bet U.S. Treasury bonds will decline. >>> Short the S&P vs Long Gold, in a 5…

I'd be more impressed with Taleb if he published the results for his Emperica fund for years other than 2008. Taleb's strategy was to buy options way out of the money. In boring years, the fund lost money. It did really well in 2008. Not enough numbers have been published to determine if his strategy was a net win over a business cycle. The win seems to have coming from the fund's short lifetime including 2008.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#66

Post by robbiep » Tue, Dec 18, 2018, 8:23 AM UTC

acidburnNSA wrote: post #39

mturmon wrote: post #25

This is an intellectually confused post. It is so unclear and muddled that only a charitable reader would suppose the author understands the mathematical and modeling issues here. I’m surprised the author published it in this form - you don’t want to jump into a controversy like this with such an unclear discussion. As just one example, the whole digression on a “decision boundary” is conceptually mistaken. Once you…

Peripherally, I like using nuclear reactor design analogies for Aleatory vs. Epistemic. Aleatory is the inherent randomness of a system. You can characterize it but it's often hard to reduce. For example, how much boron impurities are mixed into your steel at the time of manufacture. Epistemic is things that are knowable but you don't know with much certainty (because it's hard to measure), like the probability that…

That might be really useful for you, but for those of us who don’t design nuclear reactors it is utterly meaningless - what are the consequences of each of the results and how much should I care about them? I would be blown away if more than 1 in 10,000 could understand what the consequences are of your statement.

Care to share?

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#67

Post by coldtea » Tue, Dec 18, 2018, 8:27 AM UTC

hn_throwaway_99 wrote: post #37

mturmon wrote: post #25

This is an intellectually confused post. It is so unclear and muddled that only a charitable reader would suppose the author understands the mathematical and modeling issues here. I’m surprised the author published it in this form - you don’t want to jump into a controversy like this with such an unclear discussion. As just one example, the whole digression on a “decision boundary” is conceptually mistaken. Once you…

Agreed. The more I read the more I got embarrassed for the author, mainly because his points were so ridiculously flimsy. I thought this was one of the worst: > Because FiveThirtyEight only predicts probabilities, they do not ever take an absolute stand on an outcome: No ‘skin in the game’ as Taleb would say. This is not, however, something their readers follow suit on. In the public eye, they (FiveThirtyEight) are j…

>Saying "this is not something their readers follow suit on" - only if you're a reader who doesn't understand probability

Isn't that the very readership they capitalize on?

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#68

Post by tempestn » Tue, Dec 18, 2018, 8:29 AM UTC

comex wrote: post #43

chimeracoder wrote: post #38

Earlier quoted context omitted.

> 100% means the event is certain to occur Actually, 100% means the event is "almost certain". That's not an accidental choice of words; it's a technical term which conveys important meaning about how we reason about probability.

I had to look this up, so for the benefit of anyone else curious: https://en.wikipedia.org/wiki/Almost_surely TL;DR: Infinities being weird as usual.

Good read. From that page, here's an example that nicely illustrates the concept:

Imagine throwing a dart at a unit square (i.e. a square with area 1) so that the dart always hits exactly one point of the square, and so that each point in the square is equally likely to be hit.

Now, notice that since the square has area 1, the probability that the dart will hit any particular subregion of the square equals the area of that subregion. For example, the probability that the dart will hit the right half of the square is 0.5, since the right half has area 0.5.

Next, consider the event that "the dart hits a diagonal of the unit square exactly". Since the areas of the diagonals of the square are zero, the probability that the dart lands exactly on a diagonal is zero. So, the dart will almost never land on a diagonal (i.e. it will almost surely not land on a diagonal). Nonetheless the set of points on the diagonals is not empty and a point on a diagonal is no less possible than any other point: the diagonal does contain valid outcomes of the experiment.

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#69

Post by coldtea » Tue, Dec 18, 2018, 8:35 AM UTC

mturmon wrote: post #25

This is an intellectually confused post. It is so unclear and muddled that only a charitable reader would suppose the author understands the mathematical and modeling issues here. I’m surprised the author published it in this form - you don’t want to jump into a controversy like this with such an unclear discussion. As just one example, the whole digression on a “decision boundary” is conceptually mistaken. Once you…

>This is just not helpful. He has said “probability of rolling a six” twice.

For different purposes.

One is "probability of rolling a six on a standard die" -- e.g. a systemic property, where we know it's 1 in 6 but we have inherent randomness in how we roll the dice (alea in aleatory comes from the latin for dice btw, as in the famous J.Ceasar quote "alea iacta est" -- well, famous from Asterix at least).

The other is the probability of rolling a six based on what we don't know but in theory could (do we have a standard 6-sided dice? Are we asked to predict an event featuring some bizarro D&D dice we haven't seen? Is it really cubic? Have the edges been treated with a file? How about its balance?)

Re: Why we should care about the Nate Silver vs. Nassim Taleb Twitter war

#70

Post by dagw » Tue, Dec 18, 2018, 8:38 AM UTC

chimeracoder wrote: post #38

comex wrote: post #21

The article seems like an oversimplification of the underlying dispute. - It’s not necessary or even useful to define an arbitrary “decision boundary” unless you actually have to make a decision. From a Bayesian perspective, a probability stands for itself: 100% means the event is certain to occur, 50% means you have no idea, and numbers in between convey varying levels of certainty. In reality, 538’s predictions are…

> 100% means the event is certain to occur Actually, 100% means the event is "almost certain". That's not an accidental choice of words; it's a technical term which conveys important meaning about how we reason about probability.

That's a distinction without meaning when talking about a finite number of outcomes.