Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nassim Taleb: We should retire the notion of standard deviation

edge.org

Post reply on HN
Print view View original

91–100 of 254 posts

Re: Nassim Taleb: We should retire the notion of standard deviation

#91

Post by jerf »

Homunculiheaded wrote: post #38

I sometimes think that progress in the 21st century will be summed up as: "The realization that the normal distribution is not the only way to model data". Taleb's favorite topic is the "black swan event" which is something that the normal distribution, and the idea of standard deviation, don't model that well. In a normal distribution very extreme events should only happen once in the lifetime of several universes.…

This is actually what I expected to read: "The standard deviation is useful because with the average and the standard deviation, one can fully characterize a normal distribution. However, the standard deviation is less useful a statistical summary the farther away from 'normal' you get, and in reality, there is no such thing as a normal distribution, as a true normal distribution is defined on the entire real number line from negative infinity to positive infinity. Reality always provides some bound, and it's often quite distorted from Guassian. For instance, a 'normal' distribution averaging 2 with a standard deviation of 1.4, bounded by 0, is quite non-Gaussian in many important ways! (Not least of which is that you're going to have to do something to replace the missing probability...)

"People rarely check how closely their data conform to the standard distribution; indeed, many people blindly apply the standard deviation to their data regardless of its distribution! The resulting number is often more obfuscatory than helpful, to the extent that it crowded out more useful summaries.

"It's a useful metric when treated carefully, but it is rare to encounter it treated carefully. Science courses would be well-served to stop teaching it in favor of a stronger emphasis on multiple distributions. (Multiple distributions are usually touched upon, but implicitly our curricula overfavor the Gaussian distribution and end up accidentally implicitly convincing students its the only one.)"

But that's just me.

Re: Nassim Taleb: We should retire the notion of standard deviation

#92

Post by CamperBob2 »

ClementM wrote: post #13

This article is based on paper Taleb published in 2007. If you want to test yourself, submit yourself to experiment in page 3: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=970480

   A stock (or a fund) has an average return of 0%. It moves 
   on average 1% a day in absolute value; the average up move 
   is 1% and the average down move is 1%.
How does that yield an average return of 0%?

Re: Nassim Taleb: We should retire the notion of standard deviation

#93

Post by azakai »

sliverstorm wrote: post #85
azakai wrote: post #78

Earlier quoted context omitted.

>> The notion of standard deviation has confused hordes of scientists > What an assertion! It also proved to be very useful for hordes of scientists... what about some examples of confused scientists ? He is exaggerating, for sure. But the point is valid: the mean average deviation (MAD) is often very different than the standard deviation (STD), and the MAD is more intuitive, it has a natural geometrical interpretati…

the MAD is more intuitive Which one, the mean average deviation or the median average deviation?

I guess that's a matter of opinion. I personally meant the mean average deviation.

Re: Nassim Taleb: We should retire the notion of standard deviation

#94

Post by spikels »

roywei wrote: post #34

four-day returns of stock x: (-.3, .3, -.3, .3) -> MAD = 0; four-day returns of stock y: (-.5, .5, -.5, .5) -> MAD = 0.

You have calculated the mean. This is only the first step in calculating the MAD (mean absolute deviation). You then need to take the absolute value of the difference (deviation) of each datapoint from this mean - for your x you get (.3, .3, .3, .3) and for your y (.5, .5, .5, .5). Finally you take the mean of these absolute deviations to get MAD(x)=0.3 and MAD(x)=0.5.

Re: Nassim Taleb: We should retire the notion of standard deviation

#95

Post by jessaustin »

corresation wrote: post #81
marvin wrote: post #66

Earlier quoted context omitted.

What is Taleb's own black swan?

After being assigned to a currency desk, he was sitting on a lot of Eurodollar options when black Monday happened in October, 1987. http://en.wikipedia.org/wiki/Black_Monday_(1987) That unpredicted event made him and his organization loads of money, and made him a sudden seer of markets. Which is humorous because his book Fooled By Randomness is primarily a bit of a bitter tirade about how we declare people savants b…

...weird that someone would register just to post that...

The implication is obvious. 'gggggggggggggg is either an employee of yours, or is in some other situation such that she doesn't want to get on your shit list. Nevertheless, the truth must come out! Hence, anonymity.

Re: Nassim Taleb: We should retire the notion of standard deviation

#96

Post by mekael »

dxbydt wrote: post #90

The notion of area has confused hordes of scientists; it is time to retire it from common use and replace it with the more effective one of circumference. Area should be left to mathematicians, topologists and developers selling real estate. There is no scientific reason to use it in statistical investigations in the age of the computer, as it does more harm than good. Say someone just asked you to measure the area o…

Possibly the best thing I've read all week. Thank you sir.

Re: Nassim Taleb: We should retire the notion of standard deviation

#97

Post by corresation »

oskarth wrote: post #86
corresation wrote: post #81

Earlier quoted context omitted.

After being assigned to a currency desk, he was sitting on a lot of Eurodollar options when black Monday happened in October, 1987. http://en.wikipedia.org/wiki/Black_Monday_(1987) That unpredicted event made him and his organization loads of money, and made him a sudden seer of markets. Which is humorous because his book Fooled By Randomness is primarily a bit of a bitter tirade about how we declare people savants b…

He did make a lot of money in 2008 as well. That is completely aside from the fact that his points would be valid regardless of how successful his individual investments have been. I am not sure what would convince you otherwise though, since it seems like you have made up your mind already. Please loosen up on the snark.

There is no snark in my post: I am outright calling Taleb a false prophet who is guilty of exactly the things he criticizes in Fooled by Randomness.

And yes, I've seen a lot of claims of various other successes of Taleb, with a lack of any actual citations or proof at all: His own fund folded after yielding mediocre returns, and then he started consulting for a fund that seems to demonstrate a tremendous amount of bluster, with some curious and cringe-inducing ways of reporting returns, the narrative reading like the story of a gambler, telling you with great pride about the 35:1 payout on their roulette bets, skipping over the 0:37 payout on all their other bets.

I've worked with a large number of hedge funds, many of whom made a lot of money in 2008. You haven't heard a word from them because they have no need to do PR or to pitch the one bet they made right. I become suspicious of any who does.

And the danger of Taleb's lucky bet was that he is listened to as an expert, at least by some. Virtually everything he had to say about government intervention after the financial crash has been proven fundamentally wrong -- actual reality demonstrated that. Thankfully a lot of people didn't listen to him.

Re: Nassim Taleb: We should retire the notion of standard deviation

#98

Post by cdumler »

sliverstorm wrote: post #85
azakai wrote: post #78

Earlier quoted context omitted.

>> The notion of standard deviation has confused hordes of scientists > What an assertion! It also proved to be very useful for hordes of scientists... what about some examples of confused scientists ? He is exaggerating, for sure. But the point is valid: the mean average deviation (MAD) is often very different than the standard deviation (STD), and the MAD is more intuitive, it has a natural geometrical interpretati…

the MAD is more intuitive Which one, the mean average deviation or the median average deviation?

Huh? I... I don't know that. AAHAHAHAHHAHAHAhahaahahahhhhhhhhh.....

Re: Nassim Taleb: We should retire the notion of standard deviation

#99

Post by klodolph »

skybrian wrote: post #69
klodolph wrote: post #54

Earlier quoted context omitted.

> I think because it's called "standard deviation" that it sounds like the thing to use or look for. Yes! Because it's an awesome trick and lets you do good estimates on napkins. The other day I was buying lunch at a food cart and thought about how much change the food carts had to carry, as a function of how many customers they have, under the assumption that they want to be able to provide correct change to 99% of…

It's a neat math trick, but it seems more accurate to say this lets you calculate bad estimates on the back of a napkin. Unless you really think food carts carry $5000 in change. The quantitative work I do has to do with measuring latency, where the minimum, median, 90%, and 99% values are more meaningful than the mean or standard deviation. Programs typically have a best-case scenario (everything cached) and a long…

Saying that it's silly to think that food carts have $5,000 in change is unproductive because I was illustrating how the calculation works, not how the economics of food carts works, and the numbers I used for illustrative purposes were not intended to reflect reality. (1,000 customers in a day? Not likely, my guess is 200 for the busiest food carts.)

But it's good to have bad estimates, at least, it's better to have bad estimates than it is to have no estimates at all. I'm not saying that standard deviation is a substitute for more thorough analysis, just that standard deviation is an improvement over just talking about the mean.

Another example: We'd like to hire you, the mean number of hours per week you'd work is 40.

Versus:

We'd like to hire you, the mean number of hours per week you'd work is 40, and the standard deviation is 15. So your bad estimate is that you'd have two 70-hour weeks each year. But it's better than no estimate.

Re: Nassim Taleb: We should retire the notion of standard deviation

#100

Post by loganfrederick »

corresation wrote: post #81
marvin wrote: post #66

Earlier quoted context omitted.

What is Taleb's own black swan?

After being assigned to a currency desk, he was sitting on a lot of Eurodollar options when black Monday happened in October, 1987. http://en.wikipedia.org/wiki/Black_Monday_(1987) That unpredicted event made him and his organization loads of money, and made him a sudden seer of markets. Which is humorous because his book Fooled By Randomness is primarily a bit of a bitter tirade about how we declare people savants b…

After reading your two above comments, I'm not sure what your point is. You've listed a bunch of facts about Taleb that he talks about. Just because people mis-attribute his success or butcher his ideas when applying them to him is not his fault.
Post reply on HN