Live data from Hacker News

Nassim Taleb: We should retire the notion of standard deviation

edge.org

81–90 of 254 posts

Re: Nassim Taleb: We should retire the notion of standard deviation

#81
post #66

Earlier quoted context omitted.

Standard deviation tells you how volatile measurements are not what mean deviation is. Those are both very real life things just not the same thing." While I am not a Taleb fan (he had his own black swan and became exactly the reviled expert his own books warn us about), your hostility in this case seems out of place. The root of his seemingly casual contemplation is that standard deviation is simply misnamed. That t…

What is Taleb's own black swan?

After being assigned to a currency desk, he was sitting on a lot of Eurodollar options when black Monday happened in October, 1987.

http://en.wikipedia.org/wiki/Black_Monday_(1987)

That unpredicted event made him and his organization loads of money, and made him a sudden seer of markets. Which is humorous because his book Fooled By Randomness is primarily a bit of a bitter tirade about how we declare people savants because they happened to be in the right place at the right time.

Since then he gets the classic seer type treatment, for instance the other post notes that he "predicted" the financial collapse of 2008 (weird that someone would register just to post that). Yet almost everyone predicted 2008. No, seriously, the house bubble and crisis, and the probable impact on the market, was absolutely common knowledge by the time Taleb made his pronouncements, and was the endless material of virtually every financial discussion. There was absolutely nothing of worth predicted in that, but that's what you get when you got lucky once.

Re: Nassim Taleb: We should retire the notion of standard deviation

#82
post #15

Taleb has a good point about people mistakenly interpreting standard deviation (sigma) as Mean Absolute Deviation (MAD). I like that he gives some conversions (sigma ~= 1.25 * MAD, for Normal distribution). I think it's rather silly to talk about "retiring" standard deviation, but we can't blame Taleb - the publication itself posed the question "2014: What Scientific Idea is Ready for Retirement?" to various scientif…

> For example, if the data is approximately normally distributed, then there is approximately a 99.99% probability that the next data observation will be Could you sketch out why similar statements couldn't be made about MAD? My (possibly flawed) intuition is that the expected proportion of observations within n\*MAD should be similarly independent of the parameters of the normal distribution.

Re: Nassim Taleb: We should retire the notion of standard deviation

#83

Standard deviation and mean absolute deviation are both useful, but I think it's silly to suggest that we all adopt exactly one measure of variability to summarize data sets. When in doubt, make a fucking histogram.

Histograms can be misleading too, especially when the breaks are not set well.

Re: Nassim Taleb: We should retire the notion of standard deviation

#84

So first about the article: >>The notion of standard deviation has confused hordes of scientists What an assertion! It also proved to be very useful for hordes of scientists... what about some examples of confused scientists ? >>There is no scientific reason to use it in statistical investigations in the age of the computer As someone who uses it daily I am eagerly awaiting his argument. >>Say someone just asked you…

[deleted]

Re: Nassim Taleb: We should retire the notion of standard deviation

#85
post #78

So first about the article: >>The notion of standard deviation has confused hordes of scientists What an assertion! It also proved to be very useful for hordes of scientists... what about some examples of confused scientists ? >>There is no scientific reason to use it in statistical investigations in the age of the computer As someone who uses it daily I am eagerly awaiting his argument. >>Say someone just asked you…

>> The notion of standard deviation has confused hordes of scientists > What an assertion! It also proved to be very useful for hordes of scientists... what about some examples of confused scientists ? He is exaggerating, for sure. But the point is valid: the mean average deviation (MAD) is often very different than the standard deviation (STD), and the MAD is more intuitive, it has a natural geometrical interpretati…

the MAD is more intuitive

Which one, the mean average deviation or the median average deviation?

Re: Nassim Taleb: We should retire the notion of standard deviation

#86
post #66

Earlier quoted context omitted.

What is Taleb's own black swan?

After being assigned to a currency desk, he was sitting on a lot of Eurodollar options when black Monday happened in October, 1987. http://en.wikipedia.org/wiki/Black_Monday_(1987) That unpredicted event made him and his organization loads of money, and made him a sudden seer of markets. Which is humorous because his book Fooled By Randomness is primarily a bit of a bitter tirade about how we declare people savants b…

He did make a lot of money in 2008 as well. That is completely aside from the fact that his points would be valid regardless of how successful his individual investments have been.

I am not sure what would convince you otherwise though, since it seems like you have made up your mind already. Please loosen up on the snark.

Re: Nassim Taleb: We should retire the notion of standard deviation

#87
NNT is my intellectual superhero but the amount of hate he gets is tremendous.

Please understand that NNT's biggest issues are not so much with the way statistical models are applied to economics and finance, but how social scientists sometimes feel compelled to apply them to social fields as well, which is plain unscientific, dumb, and mostly disastrous.

So when you bear down on his arguments, please keep this context in mind.

Re: Nassim Taleb: We should retire the notion of standard deviation

#88

So first about the article: >>The notion of standard deviation has confused hordes of scientists What an assertion! It also proved to be very useful for hordes of scientists... what about some examples of confused scientists ? >>There is no scientific reason to use it in statistical investigations in the age of the computer As someone who uses it daily I am eagerly awaiting his argument. >>Say someone just asked you…

> The guy is a vile idiot of the worst kind: ignorant and aggressive.

Your comment is a bit irritating. I don't know the author of this blog post but I think he has a couple of valid points. You seem to dislike him for some other reasons. Feel free to share them. The blog post is still perfectly reasonable.

Anyway, I agree some of your comments as well, namely this:

> I can give hundreds when std dev is useful and mean deviation isn't. Anything when you decide what % of yoru bankroll to bet on perceived edge for example.

and this:

> Guess what though, taking the squares have a purpose: it penalizes big deviations so two situations which have the same mean deviation but one is more stable have different standard deviations.

But I don't see why this comment from N.T. is stupid:

>>>It is all due to a historical accident: in 1893, the great Karl Pearson introduced the term "standard deviation" for what had been known as "root mean square error". The confusion started then: people thought it meant mean deviation.

I actually agree that standard deviation is a confusing name, why do you think that's a stupid comment?

Re: Nassim Taleb: We should retire the notion of standard deviation

#89

So first about the article: >>The notion of standard deviation has confused hordes of scientists What an assertion! It also proved to be very useful for hordes of scientists... what about some examples of confused scientists ? >>There is no scientific reason to use it in statistical investigations in the age of the computer As someone who uses it daily I am eagerly awaiting his argument. >>Say someone just asked you…

[deleted]

Re: Nassim Taleb: We should retire the notion of standard deviation

#90
The notion of area has confused hordes of scientists; it is time to retire it from common use and replace it with the more effective one of circumference. Area should be left to mathematicians, topologists and developers selling real estate. There is no scientific reason to use it in statistical investigations in the age of the computer, as it does more harm than good.

Say someone just asked you to measure the area of a circle with radius pi. The area is exactly 31. But how do you do it?

scala> math.round(math.Pi * math.Pi * math.Pi).toInt

res1: Int = 31

Do you pack the circle with n people, count them up and verify n == 31 ? Or do you pour a red liquid into the circle and fill it up, then drain it and measure the amount of red ? For there are serious differences between the two methods.

If instead, you were asked to measure the circumference of a circle with radius pi.

scala> math.round(2 * math.Pi * math.Pi).toInt

res2: Int = 20

You just ask an able-bodied man, perhaps an unemployed migrant, to walk around this circle while another man, an upstanding Stanford sophomore, starts walking from Stanford to meet his maker, I mean VC, well its the same thing...

So by the time the migrant finishes walking around the circle, our upstanding Stanford entrepreneur is greeting the VC on the tarmac of the San Francisco International Airport. This leads one to rightfully believe that the circumference of the circle of radius pi is exactly the distance from Stanford to the SF Airport ie. 20 miles. It corresponds to "real life" much better than the first—and to reality. In fact, whenever people make decisions after being supplied with the area, they act as if it were the distance from their university to the airport.

It is all due to a historical accident: in 250BC, the Greek mathematician Archimedes introduced Prop 2, the Prevention of Farm Cruelty Act ( http://en.wikipedia.org/wiki/California_Proposition_2_(2008) ). No I believe this was a different Prop 2. This Prop 2 states that the area of a circle is to the square on its diameter as 11 to 14 (http://en.wikipedia.org/wiki/Measurement_of_a_Circle ) .The confusion started then: people thought it meant areas had to do with being cruel to farm animals. But it is not just journalists who fall for the mistake: I recall seeing official documents from the department of data scientists, which found that a high number of data scientists (many with PhDs) also get confused in real life.

It all comes from bad terminology for something non-intuitive. Despite this confusion, Archimedes persisted in the folly by drawing circles in the sand, an infantile persuasion, surely. When the Romans waged war, Archimedes was still computing the area of the circle. The Roman soldier asked him to step outside, but Archimedes exclaimed "Do not disturb my circles!" (http://en.wikipedia.org/wiki/Noli_turbare_circulos_meos)

He was rightfully executed by the soldier for this grievous offense. It is sad that such a minor mathematician can lead to so much confusion: our scientific tools are way too far ahead of our casual intuitions, which starts to be a problem with a mad Greek. So I close with a statement by famed rapper Sir Joey Bada$$, extolling the virtues of the circumference: "So I keep my circumference of deep fried friends like dumplings, But fuck that nigga we munching, we hungry." (http://rapgenius.com/1931938/Joey-bada-hilary-swank/So-i-kee...)

Post reply on HN