Nassim Taleb: We should retire the notion of standard deviation • Hacker News Board

Re: Nassim Taleb: We should retire the notion of standard deviation

#191

Post by klodolph » Thu, Jan 16, 2014, 12:45 AM UTC

skybrian wrote: post #154

klodolph wrote: post #99

Earlier quoted context omitted.

Saying that it's silly to think that food carts have $5,000 in change is unproductive because I was illustrating how the calculation works, not how the economics of food carts works, and the numbers I used for illustrative purposes were not intended to reflect reality. (1,000 customers in a day? Not likely, my guess is 200 for the busiest food carts.) But it's good to have bad estimates, at least, it's better to have…

Sure, two points is better than one but what's special about two? I'd rather have a graph. We have computers so there's rarely a reason to compress the data so much.

We often have to compress the data down to a single decision or statistic: yes/no should I accept the job offer, how much money should I save before buying a house, or what's the probability that I'll die in the next 10 years.

I hate to quote XKCD, but it's like saying your favorite map projection is a globe (http://xkcd.com/977/). Yes, you've preserved all the data, but even with computers, your beloved graph will not make it all the way to the end.

Re: Nassim Taleb: We should retire the notion of standard deviation

#192

Post by yen223 » Thu, Jan 16, 2014, 12:45 AM UTC

sliverstorm wrote: post #40

bluecalm wrote: post #16

So first about the article: >>The notion of standard deviation has confused hordes of scientists What an assertion! It also proved to be very useful for hordes of scientists... what about some examples of confused scientists ? >>There is no scientific reason to use it in statistical investigations in the age of the computer As someone who uses it daily I am eagerly awaiting his argument. >>Say someone just asked you…

The funny thing is I've never used absolute deviation before, so I'm slightly confused by absolute deviation right now as I wrap my head around what it means relative to sigma (which I have used, lightly, for years) and why anyone would want to use it instead... edit: apparently MAD can refer to either "mean absolute deviation" or "median absolute deviation". Yup, this sure isn't going to confuse anybody.

MAD can also refer to Mutually Assured Destruction, which seems fitting given this thread.

Re: Nassim Taleb: We should retire the notion of standard deviation

#193

Post by jessaustin » Thu, Jan 16, 2014, 12:53 AM UTC

notastartup wrote: post #112

I've been a long time fan of Dr. Nassim Taleb. First book I've read was the one about his time as a day trader and how on the Black Friday market crash, he made a killing and cleared his desk and never had to work again. There are those that dislike his ideas because it is threatening to their existing assumptions about probability and statistics. He argues that experts and majority of people do not account for the u…

I think I agree with most of this, but now aren't we supposed to say that the drop in crime is due more to reduced childhood lead exposures than to abortion? Not just because it's a feel-good, humanistic explanation rather than a horrific racist eugenics explanation, but also because recent studies show a pretty strong lagged correlation...

Re: Nassim Taleb: We should retire the notion of standard deviation

#194

Post by baking » Thu, Jan 16, 2014, 12:54 AM UTC

ClementM wrote: post #13

This article is based on paper Taleb published in 2007. If you want to test yourself, submit yourself to experiment in page 3: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=970480

Yes, start by writing a confusing question. One that starts talking about average up moves and average down moves and then switches to asking about the standard deviation of moves. Then publish a paper showing that people were confused by your question. Now you have "research" to back up your claim that everyone is confused by "mean deviation" and "standard deviation".

Re: Nassim Taleb: We should retire the notion of standard deviation

#195

Post by [deleted] » Thu, Jan 16, 2014, 12:59 AM UTC

tsax wrote: post #133

antiterra wrote: post #56

Earlier quoted context omitted.

The guy is clearly not an idiot in the general sense; I bet he is of, at least, average intelligence. I also have my doubts that he is 'vile' for the majority of connotations and denotations of that word. Because of this, just shouting 'he's an idiot!' is vulnerable to being dismissed as hyperbole, and your baby gets thrown out with the bathwater. You seem exasperated at his condescending attitude, but make no effort…

Yeah a polyglot who writes classical Greek, Arabic, French, English and can do advanced statistical modeling is clearly an idiot right?

[deleted]

Re: Nassim Taleb: We should retire the notion of standard deviation

#196

Post by baking » Thu, Jan 16, 2014, 1:00 AM UTC

CamperBob2 wrote: post #92

ClementM wrote: post #13

This article is based on paper Taleb published in 2007. If you want to test yourself, submit yourself to experiment in page 3: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=970480

A stock (or a fund) has an average return of 0%. It moves on average 1% a day in absolute value; the average up move is 1% and the average down move is 1%. How does that yield an average return of 0%?

The usual way to do this is to take the natural log, so an up 1% day followed by a down 1% day (or vice versa) will always net out to a 0% change.

Re: Nassim Taleb: We should retire the notion of standard deviation

#197

Post by klodolph » Thu, Jan 16, 2014, 1:09 AM UTC

ced wrote: post #173

klodolph wrote: post #148

Earlier quoted context omitted.

> and the MAD is more intuitive, it has a natural geometrical interpretation It's less intuitive, not more intuitive, at least for me. And the standard deviation definitely has a more geometric interpretation than MAD. If you measure a hundred samples, and you want to figure out how much they differ from the expected values, what could be more intuitive than Euclidean distance? But most people never bother to try and…

Maybe I misunderstood what you meant, but I would say that the L_2 norm amounts to MAD. L2 means sqrt(x²+y²+...) and in 1D, that becomes sqrt(x²) = |x|

In 1D, L_2 = L_1. But if you are talking about 100 sample points, the data has 100 dimensions.

Re: Nassim Taleb: We should retire the notion of standard deviation

#198

Post by klodolph » Thu, Jan 16, 2014, 1:13 AM UTC

relate wrote: post #186

klodolph wrote: post #148

Earlier quoted context omitted.

> and the MAD is more intuitive, it has a natural geometrical interpretation It's less intuitive, not more intuitive, at least for me. And the standard deviation definitely has a more geometric interpretation than MAD. If you measure a hundred samples, and you want to figure out how much they differ from the expected values, what could be more intuitive than Euclidean distance? But most people never bother to try and…

This makes no sense. If you want to know how much they differ from the expected value, you first define what you mean by difference (i.e. L1 or L2 norm) and then measure it somehow. The standard deviation is an estimator for the square root of the expected value of this difference, when it's chosen to be the SQUARED L_1 distance (which is the same as the L_2 distance in 1-D). The MAD takes the mean of the L-1 distanc…

> that's not an intuitive expression based on the problem statement

That's the danger I'm talking about when you start using the word "intuitive". Intuition is relative, and someone who works with mathematics or statistics will develop a mathematical intuition about things. Just like if you're an experienced driver you'll intuitively know when other drivers are about to change lanes, even before they signal.

I think of L_2 more intuitive because it physical space uses the L_2 norm.

The other thing that makes L_1 counterintuitive is that if you measure the absolute deviation from the mean, then you aren't minimizing the deviation—in order to do that, you have to choose the median.

In other words, you say "this is the center" and "this is the measure of how far away everything is from the center", but you could have picked a different center which has a lower distance from your data. Counterintuitive.

Re: Nassim Taleb: We should retire the notion of standard deviation

#199

Post by lhnz » Thu, Jan 16, 2014, 1:26 AM UTC

Fomite wrote: post #109

tel wrote: post #77

Earlier quoted context omitted.

He's inflammatory. Really, really inflammatory. He also tends to write about obvious technical points so if you do pick through the morass and find the gem you'll be a bit underwhelmed. In this instance, standard deviations are obviously important statistics to compute and have many, many useful properties, sure, but Taleb rightly points out that they can be misused and I agree with him that their misuse is endemic i…

I think that's my problem. By the time I've waded through the insufferable smugness, the anecdotes and the random and meaningless attention to things I've never encountered in practice, what is left is interesting, but not nearly worth the effort.

What is worth the effort to you?

I have found his books more stimulating than Hacker News for example...

Re: Nassim Taleb: We should retire the notion of standard deviation

#200

Post by thaumasiotes » Thu, Jan 16, 2014, 1:27 AM UTC

azakai wrote: post #78

bluecalm wrote: post #16

So first about the article: >>The notion of standard deviation has confused hordes of scientists What an assertion! It also proved to be very useful for hordes of scientists... what about some examples of confused scientists ? >>There is no scientific reason to use it in statistical investigations in the age of the computer As someone who uses it daily I am eagerly awaiting his argument. >>Say someone just asked you…

>> The notion of standard deviation has confused hordes of scientists > What an assertion! It also proved to be very useful for hordes of scientists... what about some examples of confused scientists ? He is exaggerating, for sure. But the point is valid: the mean average deviation (MAD) is often very different than the standard deviation (STD), and the MAD is more intuitive, it has a natural geometrical interpretati…

> The STD is far more easy to analyze in a mathematical way. That is the huge value it brings

This is also what I have read (I have essentially no experience with statistics).

> squaring is an operation you can take the derivative of, but absolute value you cannot

We can talk without wild, obviously false hyperbole. It's trivial to take the derivative of the absolute value function. Working with it is more difficult, but taking the derivative is as easy as anything in math:

    f'(x) = -1 (when x  0)