Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nassim Taleb: We should retire the notion of standard deviation

edge.org

Post reply on HN
Print view View original

211–220 of 254 posts

Re: Nassim Taleb: We should retire the notion of standard deviation

#211

Post by thaumasiotes »

photojosh wrote: post #206
thaumasiotes wrote: post #200

Earlier quoted context omitted.

> The STD is far more easy to analyze in a mathematical way. That is the huge value it brings This is also what I have read (I have essentially no experience with statistics). > squaring is an operation you can take the derivative of, but absolute value you cannot We can talk without wild, obviously false hyperbole. It's trivial to take the derivative of the absolute value function. Working with it is more difficult,…

That little discontinuity at the origin is kind of a pain though.

Oh, completely agreed. I was enlightened when I read that standard deviations were defined the way they were in order to make analysis easier; it seems more than reasonable to me. But there's no difficulty in just taking the derivative of |x|, and I hate seeing random misinformation out there.

http://xkcd.com/386/

Re: Nassim Taleb: We should retire the notion of standard deviation

#212

Post by michaelhoffman »

randomsample2 wrote: post #108
michaelhoffman wrote: post #83

Earlier quoted context omitted.

Histograms can be misleading too, especially when the breaks are not set well.

Anything can be misleading, but in most cases a histogram conveys more information than a single number ever could.

Absolutely. They just aren't a panacea.

Re: Nassim Taleb: We should retire the notion of standard deviation

#213

Post by bayesianhorse »

kylebrown wrote: post #210
bayesianhorse wrote: post #179

Nassim Taleb somehow likes to beat up on normals... We Bayesians have similar notions, but we usually try not to overly bully frequentist methods, the poor things. Also, being familiar with Bayesian methods, a lot of what Taleb is saying sounds vaguely familiar...

Well, frequentist methods are taking the blame for the most recent financial crisis, such as assuming a normal distribution when the empirical one is fat-tailed. Perhaps the Bayesian methods will take the blame in the next financial crisis. Such as the error in estimating a non-stationary distribution and quantifying the uncertainty.

Statistical economists or analysts always said that the normal distribution is a simplification and that this simplification has its own problems.

It's just that traders and bankers would fire statisticians who were too vocal about it as wasting their time with unnecessary explanations...

Bayesian methods in general can only take the blame if you can prove some other method being more reliable.

Also they have another advantage: Bayesian methods are so mind blowing and beautiful, that it is hard to blame them for anything!

Re: Nassim Taleb: We should retire the notion of standard deviation

#214

Post by skybrian »

klodolph wrote: post #191
skybrian wrote: post #154

Earlier quoted context omitted.

Sure, two points is better than one but what's special about two? I'd rather have a graph. We have computers so there's rarely a reason to compress the data so much.

We often have to compress the data down to a single decision or statistic: yes/no should I accept the job offer, how much money should I save before buying a house, or what's the probability that I'll die in the next 10 years. I hate to quote XKCD, but it's like saying your favorite map projection is a globe ( http://xkcd.com/977/ ). Yes, you've preserved all the data, but even with computers, your beloved graph will…

Preserving all the data is the logical endpoint but that's not what I was suggesting. I'm just saying there's nothing special about keeping two points.

I'd rather not feed two points to my decision algorithm, whether it's machine learning or a human looking at the data. It makes more sense to make some attempt to preserve the shape of the graph unless you have strong reason to believe it's Gaussian, and even then the assumption should be checked.

Re: Nassim Taleb: We should retire the notion of standard deviation

#215

Post by TTPrograms »

There is some argument that MAD is actually better than RMS for a lot of applications. Apparently it predated RMS, but one of the reasons it was switched to was because RMS minimizing linear regression is much, much simpler to calculate. Also consider comparing the robustness of RMS based regression with MAD based regression. See: http://matlabdatamining.blogspot.com/2007/10/l-1-linear-regr...

Re: Nassim Taleb: We should retire the notion of standard deviation

#216

Post by hrjet »

photojosh wrote: post #208
azakai wrote: post #140

Earlier quoted context omitted.

> The standard deviation maps to processes where the "importance" of changes is proportional to their square. Sure, but the point of the article is that STD is used very commonly, in places where that does not make sense. For example, it is common to see things like "the weight of the test subjects was 170cm (STD 5cm)".

Indeed, why does it not make sense? I read that, I immediately know that about 68% will be between 165 and 170 cm. http://en.wikipedia.org/wiki/68–95–99.7_rule

That rule only applies to normal distribution. So you would have made an assumption.

Re: Nassim Taleb: We should retire the notion of standard deviation

#217

Post by yetanotherphd »

I had hoped this would be about the revolution occurring in statistics/econometrics where confidence intervals based on strong parametric assumptions (e.g. the confidence intervals you would obtain using the standard deviation) are being replaced by confidence intervals obtained using the bootstrap (and other non-parametric methods) that don't rely on such strong assumptions.

But no, it is just advocating using Mean absolute distance instead of the standard deviation. Which I guess is to be expected from someone whose work focuses mostly on long-tailed distributions.

Still, I think that non-parametric methods are much more valuable as a solution to dealing with non-normal data than what Taleb is proposing.

Re: Nassim Taleb: We should retire the notion of standard deviation

#218

Post by arjie »

avenger123 wrote: post #169
edw519 wrote: post #102

Earlier quoted context omitted.

Counterexample: | .. | . . | waste | . . | stuff | of | . . | I | my | . . | don't | time | . stuff I'll . | grok | | . read . | |(Nassim Taleb) | | . | . | | . |______________________________________________________ -3 -2 -1 0 1 2 3 standard deviations

Very nice! I did have his books on my reading list. But this whole discussion definitely puts me closer to the "waste of my time" side of the distribution.

You should be aware that HN and other online comment forums often have 'negative' posts. Comments that attempt to show up the author of a linked article are a kind of intellectual one-upmanship that is common on Reddit/HN/etc.

I haven't read Nassim Taleb's work either, but comment forums like HN are a poor place to get an idea of the quality of someone's work unless they have clear and substantial criticisms.

You're usually better off getting your reviews elsewhere.

Re: Nassim Taleb: We should retire the notion of standard deviation

#219

Post by avenger123 »

lhnz wrote: post #203
avenger123 wrote: post #169

Earlier quoted context omitted.

Very nice! I did have his books on my reading list. But this whole discussion definitely puts me closer to the "waste of my time" side of the distribution.

That would be a mistake. He's getting a lot of hate because he's insulted a lot of people while stretching some of his arguments and opinions too far, but his ideas are still better intellectual reading than most of what's available elsewhere. If you care to read the comments again, you'll notice many of the disagreements with him are on the level of knee-jerk emotional reactions and not well reasoned criticism. Howe…

+1 for this. Your comment is definitely the most reasonable in this discussion. I'll likely read his Fooled by Randomness and Black Swan at some point.

Re: Nassim Taleb: We should retire the notion of standard deviation

#220

Post by robotcookies »

corresation wrote: post #81
marvin wrote: post #66

Earlier quoted context omitted.

What is Taleb's own black swan?

After being assigned to a currency desk, he was sitting on a lot of Eurodollar options when black Monday happened in October, 1987. http://en.wikipedia.org/wiki/Black_Monday_(1987) That unpredicted event made him and his organization loads of money, and made him a sudden seer of markets. Which is humorous because his book Fooled By Randomness is primarily a bit of a bitter tirade about how we declare people savants b…

Actually, Taleb has explicitly stated that the 2008 financial crisis was not a Black Swan nor does he claim to be able to predict any of them.
Post reply on HN