Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unlearning descriptive statistics

debrouwere.org

Post reply on HN
Print view View original

31–40 of 95 posts

Re: Unlearning descriptive statistics

#31

Post by srean »

somestag wrote: post #15

I think the message of the article is great: move beyond the "standard" descriptions and pay more attention to what you're trying to show and who your audience is. That said, it's a slight pet peeve of mine when people recommend the median over the mean to describe center. The median, on its own, does not describe what is "typical" any more than the mean does; it just has a small advantage in that it will always map…

> when people recommend the median over the mean to describe center. The median, on its own, does not describe what is "typical" any more than the mean does; it just has a small advantage

I think this swings the other extreme in selling the 'median' short. As long as we agree that it is only strictly meaningful to talk about the 'center' for symmetric distributions, median does a fine job.

In fact in many realistic scenarios a far better job than the mean. The main trouble is the normal or the Gaussian distribution is no where close to being as ubiquitous as it seems, neither is CLT as universal as it is made out to be. Gauss sort of got away with it, Gauss did not discover the distribution nor the associated CLT.

Many real data of day to day consequence have heavy tails, and mean is a pathetic measure of 'central tendency' for these. Mean is particularly sensitive to outliers. Median does significantly better than the mean in this non-academic situation. Although one could do better than median for symmetric heavy tailed data (for example trimmed means), but its mean that I find guilty of entirely disproportionate fame. If its needed to exaggerate median a bit to get people to grow up beyond the pervasive Normal / Gaussian fetish, I am all for it.

No single number is going to characterize what is 'typical'. One really needs the CDF here, and yes avoid estimating densities as much as possible.

BTW mapping to a real observation is not true, you only get a 50% chance of that.

Re: Unlearning descriptive statistics

#32

Post by FabHK »

apathy wrote: post #16
shmageggy wrote: post #9

> But why do you want a number at all? Because lots of data analysis questions hinge upon the association between two data sets, and it's nice (crucial) to be able to quantify this value. Especially because > While statisticians are generally quite good at estimating a correlation from a picture and vice versa, most people are not. The author says > Still not happy and absolutely want a number? You would do well to s…

1) it is pretty amazing that normit transformations (map the quantiles of a non-normal distribution onto a Gaussian and use that) don't seem to be on this guy's radar. We use distributions with linearly additive and affine invariant properties (normal plus normal is normal, bernoulli plus bernoulli is bitwise bernoulli) because we find linear algebra very useful. Nonparametric tests and procedures erode your power; n…

Several great points (normit is the basis for the Gaussian copula, which was used to great effect to model the CDOs (collateralised debt obligations) that blew up in the GFC (global financial crisis)); but it would have been possible to raise them while being less dismissive...

Re: Unlearning descriptive statistics

#33

Post by amelius »

somestag wrote: post #15

I think the message of the article is great: move beyond the "standard" descriptions and pay more attention to what you're trying to show and who your audience is. That said, it's a slight pet peeve of mine when people recommend the median over the mean to describe center. The median, on its own, does not describe what is "typical" any more than the mean does; it just has a small advantage in that it will always map…

Is it possible to sensibly generalize the concept of median to "higher orders"?

E.g., I can imagine the difference between the 25th and 75th percentile to be descriptive of spread (like standard deviation), but those numbers seem arbitrary.

Re: Unlearning descriptive statistics

#34

Post by srean »

apathy wrote: post #16
shmageggy wrote: post #9

> But why do you want a number at all? Because lots of data analysis questions hinge upon the association between two data sets, and it's nice (crucial) to be able to quantify this value. Especially because > While statisticians are generally quite good at estimating a correlation from a picture and vice versa, most people are not. The author says > Still not happy and absolutely want a number? You would do well to s…

1) it is pretty amazing that normit transformations (map the quantiles of a non-normal distribution onto a Gaussian and use that) don't seem to be on this guy's radar. We use distributions with linearly additive and affine invariant properties (normal plus normal is normal, bernoulli plus bernoulli is bitwise bernoulli) because we find linear algebra very useful. Nonparametric tests and procedures erode your power; n…

I largely agree with what you are saying but estimating the population transformation that makes the transformed data Gaussian from a finite sample is far from trivial.

If you have any pointers to results that show distribution free guarantee of increased power I would be super happy to read.

Here's a question for you , why not just deal with the quantiles directly (for example with quantile regression for regression tasks) and not map it to the quantiles of a Gaussian ?

Re: Unlearning descriptive statistics

#35

Post by FabHK »

jfaucett wrote: post #3

This is a great read. I've been thinking alot about these basic concepts myself lately, especially the idea of central tendency. Essentially, we can make up any method we want to summarize data and give us a single value that represents the central location of the data i.e. mean vs. least absolute distance vs. distance squared, etc. I haven't thought much about the difference between "typical case" or "expected value…

One good way to think about it is to ask what norm ("distance") between all the datapoints and your "central tendency" statistic you want to minimize, as the article alluded to.

For L2 (squared distance), you get the mean: For fixed x_i,

sum of i=1..N of (x_i - M)^2

is smallest for M = mean.

For L1 (absolute distance), you get the median.

For L0 ("identical or not"), you get the mode.

You can come up with other and alternative concepts, but it's kind of neat that the 3 most common descriptions of central tendency pop out of this one unified approach.

Re: Unlearning descriptive statistics

#36

Post by eanzenberg »

minimaxir wrote: post #2

It's worth noting that most statistical packages include the ability to generate a five-number summary ( https://en.m.wikipedia.org/wiki/Five-number_summary ), which generates many of the described skew-independent descriptive statistics automatically. (R, for example, will generate a five-number summary for each feature of a data frame with a simple summary(df) ) The TL;DR is to just plot everything if possible to v…

Pandas has a .describe() function on dataframes too.

Re: Unlearning descriptive statistics

#37

Post by eanzenberg »

somestag wrote: post #15

I think the message of the article is great: move beyond the "standard" descriptions and pay more attention to what you're trying to show and who your audience is. That said, it's a slight pet peeve of mine when people recommend the median over the mean to describe center. The median, on its own, does not describe what is "typical" any more than the mean does; it just has a small advantage in that it will always map…

I think the idea is to move away from worshiping summary statistics such as mean or median and starting to look at differences of distributions.

Re: Unlearning descriptive statistics

#39

Post by eanzenberg »

apathy wrote: post #16
shmageggy wrote: post #9

> But why do you want a number at all? Because lots of data analysis questions hinge upon the association between two data sets, and it's nice (crucial) to be able to quantify this value. Especially because > While statisticians are generally quite good at estimating a correlation from a picture and vice versa, most people are not. The author says > Still not happy and absolutely want a number? You would do well to s…

1) it is pretty amazing that normit transformations (map the quantiles of a non-normal distribution onto a Gaussian and use that) don't seem to be on this guy's radar. We use distributions with linearly additive and affine invariant properties (normal plus normal is normal, bernoulli plus bernoulli is bitwise bernoulli) because we find linear algebra very useful. Nonparametric tests and procedures erode your power; n…

Re: the linear methods vs. neural networks

It depends on the domain. Logistic or most other classifiers won't get close to NN when classifying images or text. It's not 80-90% of the power.

You are right when dealing with data that is not highly-dimensional and not very non-linear either. Also plenty of other domains..

Re: Unlearning descriptive statistics

#40

Post by zeteo »

somestag wrote: post #15

I think the message of the article is great: move beyond the "standard" descriptions and pay more attention to what you're trying to show and who your audience is. That said, it's a slight pet peeve of mine when people recommend the median over the mean to describe center. The median, on its own, does not describe what is "typical" any more than the mean does; it just has a small advantage in that it will always map…

>The median [...] just has a small advantage in that it will always map a real observation, so for discrete data you don't end up with things like "1.9 legs."

If you listen to someone like Taleb, the main advantage they mention for the median is that it's a more robust statistic. For fat-tailed distributions the average can jump all over as new data come in.

Post reply on HN