Live data from Hacker News

Unlearning descriptive statistics

debrouwere.org

11–20 of 95 posts

Re: Unlearning descriptive statistics

#11
post #4

It's worth noting that most statistical packages include the ability to generate a five-number summary ( https://en.m.wikipedia.org/wiki/Five-number_summary ), which generates many of the described skew-independent descriptive statistics automatically. (R, for example, will generate a five-number summary for each feature of a data frame with a simple summary(df) ) The TL;DR is to just plot everything if possible to v…

you can also use the aptly named fivenum() func in R :)

summary(df) will do that for every column of a dataframe with less code... which is why few people use fivenum()

The majority of defaults in R exist because statisticians do these things all the time and don't like typing any more than programmers. Time spent typing could be spent thinking about what assumptions are being violated and whether the results offer any useful insight.

Re: Unlearning descriptive statistics

#12
This is terrific.

In the field of web ops, I've had great success tracking the 95th percentile of request time rather than the mean, median, mode, or any other descriptive metric.

The systems I worked with were, like many systems, ordinarily very efficient. That meant the mean and median metrics washed away the occasional troublesome request and hid it from the metrics dashboard.

But knowing the 95th percentile was out of bounds allowed my team to investigate and try to remediate the trouble spots.

This was especially useful in troubleshooting fax delivery (yeah, I know, I know: it was healthcare stuff; fax is considered secure and email is not) through unbelievably flakey private branch exchanges (big city hospitals)

Re: Unlearning descriptive statistics

#13

For readers who are OK with some math, I recommend John Myles White's eye-opening post about means, medians, and modes: http://www.johnmyleswhite.com/notebook/2013/03/22/modes-medi... He describes these summary descriptive stats in terms of what penalty function they minimize: mean minimizes L2, median minimizes L1, mode minimizes L0. A single-number statistic is _going_ to leave things out, so if you must boil thing…

Glad you enjoyed that post so much. It really is a shame that we do such a bad job of teaching students about the inherent subjectivity of descriptive statistics and let students leave their courses with dangerous ideas about the existence of a Holy Grail statistic that will solve all of their problems.

Re: Unlearning descriptive statistics

#14

> But why do you want a number at all? Because lots of data analysis questions hinge upon the association between two data sets, and it's nice (crucial) to be able to quantify this value. Especially because > While statisticians are generally quite good at estimating a correlation from a picture and vice versa, most people are not. The author says > Still not happy and absolutely want a number? You would do well to s…

> OK, so what else do you suggest?

Slopes, a.k.a. the parameters in a regression analysis, ideally as a confidence interval or prediction interval to account for uncertainty. Interpreting and communicating regression analyses is a pretty big topic on its own so I chose to only hint at it, though I understand that might not be very satisfying for some readers.

Re: Unlearning descriptive statistics

#15
I think the message of the article is great: move beyond the "standard" descriptions and pay more attention to what you're trying to show and who your audience is.

That said, it's a slight pet peeve of mine when people recommend the median over the mean to describe center. The median, on its own, does not describe what is "typical" any more than the mean does; it just has a small advantage in that it will always map a real observation, so for discrete data you don't end up with things like "1.9 legs." (That said, a mean of 1.9 legs actually seems much more informative to me than a median of 2 legs, so even in that case I prefer the mean.) It's easy to envision many situations where the median's representation is wildly inaccurate, just as you can imagine ways in which the mean can be misleading.

The median is also insensitive to skew, which is often cited as a good thing, but really that's something you should determine on a case-by-case basis. In many (most?) applications, there's no tangible benefit in having a measure of center that ignores skew. The median's insensitivity also creates strange situations where subgroups in the population end up unrepresented (e.g. if the poorest 20% become even poorer because of changes to the tax code, the median doesn't budge). In general, the median conveys remarkably little information. It's great at showing the center, but it gives zero indication of anything else. The mean certainly has its issues, but it's far from inferior. Use the median when the situation calls for it, certainly, but realize that it's just as limited as any other measure, if not more so.

Re: Unlearning descriptive statistics

#16

> But why do you want a number at all? Because lots of data analysis questions hinge upon the association between two data sets, and it's nice (crucial) to be able to quantify this value. Especially because > While statisticians are generally quite good at estimating a correlation from a picture and vice versa, most people are not. The author says > Still not happy and absolutely want a number? You would do well to s…

1) it is pretty amazing that normit transformations (map the quantiles of a non-normal distribution onto a Gaussian and use that) don't seem to be on this guy's radar. We use distributions with linearly additive and affine invariant properties (normal plus normal is normal, bernoulli plus bernoulli is bitwise bernoulli) because we find linear algebra very useful. Nonparametric tests and procedures erode your power; normit transformations usually increase it. I did part of my dissertation on this; it's partly to do with asymptotics, but also partly due to the robustness of Gaussian error assumptions thanks to the CLT.

I realized recently that a lot of the trouble people have with training neural networks stems from their lack of training in basic model evaluation. If you stack a bunch of shitty penalized regressions (which is what linear/logistic + relU hinge loss represents) you now have one gigantic shitty regression which is harder to debug. If your early steps are thrown out of whack by outliers, your later steps will be too. Dropout is an attempt to remedy this, but you tend to lose power when you shrink your dataset or model, so (per usual) there really is no such thing as a free lunch. But most of the tradeoffs make more sense when you are able to evaluate each layer as a predictor/filter. Scaling this up to deep models is hard, therefore debugging deep models is hard. Not exactly a big leap.

There is a reason people say "an expert is a master of the fundamentals". Building a castle on a swamp gives poor results. If you can't design an experiment to test your model and its assumptions, your model will suck. This is not rocket surgery. A GPU allows you to make more mistakes, faster, if that's what you want. If you have the fundamentals nailed down, and enough data to avoid overfitting, then nonlinear approaches can be incredibly powerful.

Most of the time a simple logistic regression will offer 80-90% of the power of a DNN, a kernel regression will offer 80-90% of the power of a CNN, and an HMM or Kalman filter will offer 80-90% of the power of an RNN. It's when you need that 10-20% "extra" to compete, and have the data to do it, that deeper or trickier architectures help.

If you can transform a bunch of correlated data so that it is 1) decorrelated and 2) close enough to Gaussian for government work, you suddenly get a tremendous amount of power from linear algebra and differential geometry "for free". This is one reason why Bayesian and graphical hierarchical mixed models work well -- you borrow information when you don't have enough to feed the model, and if you have some domain expertise, this allows you to keep the model from making stupid or impossible predictions.

Anyways. I have had fun lately playing with various deep, recurrent, and adversarial architectures. I don't mean to imply they aren't tremendously powerful in the right hands. But so is a Hole Hawg. Don't use a Hole Hawg when a paper punch is all you really need.

2) What (good) statisticians excel at is catching faulty assumptions. (I'll leave it to the reader to decide whether this data-scientist-for-hire has done a good job of that in his piece) So we plot our data, marginally or via projections, all the damned time. If you don't, sooner or later it will bite you in the ass, and then you too can join the ranks of the always-plotting. However, choosing which margins or conditional distributions to plot in a high-dimensional or sparse dataset is important to avoid wasting a lot of time. So whether via transformation or penalization (e.g. graphical lasso) or both, we usually try to prune things down and home on on "the good stuff". Prioritizing what to do first is most easily done if you have a number and can rank the putative significance by that number. Use Spearman, use energy statistics (distance correlation), use marginal score tests -- IDGAF, just use these as guidelines and plot the damned data.

Corollary: if someone shows you fancy plots and never simple ones containing clouds of individual data points, they're probably full of shit. Boxplots should be beeswarms, loess plots should have scatterplots (smoothed or otherwise) behind them. And for god's sake plot your residuals, either implicitly or explicitly.

3) see above. The author is good at fussing, and brings up some classical points. But they're not really his points. Median and MAD are more robust to outliers than mean and standard deviation, but that makes them less sensitive, too. Check your assumptions, plot everything, use the numbers as advisory quantities rather than final results.

Re: Unlearning descriptive statistics

#17

For readers who are OK with some math, I recommend John Myles White's eye-opening post about means, medians, and modes: http://www.johnmyleswhite.com/notebook/2013/03/22/modes-medi... He describes these summary descriptive stats in terms of what penalty function they minimize: mean minimizes L2, median minimizes L1, mode minimizes L0. A single-number statistic is _going_ to leave things out, so if you must boil thing…

I always pull out Anscombe's Quartet https://en.wikipedia.org/wiki/Anscombe's_quartet The four datasets have the same mean, variance, and linear regression line, but are very different from one another.

Re: Unlearning descriptive statistics

#19

I think the message of the article is great: move beyond the "standard" descriptions and pay more attention to what you're trying to show and who your audience is. That said, it's a slight pet peeve of mine when people recommend the median over the mean to describe center. The median, on its own, does not describe what is "typical" any more than the mean does; it just has a small advantage in that it will always map…

Great points. Discrete data with few categories are a good case where the median might not cut it. As a contrived example..

data: 0 0 0 1 1

mean: 2 / 5

median: 0

Re: Unlearning descriptive statistics

#20

For readers who are OK with some math, I recommend John Myles White's eye-opening post about means, medians, and modes: http://www.johnmyleswhite.com/notebook/2013/03/22/modes-medi... He describes these summary descriptive stats in terms of what penalty function they minimize: mean minimizes L2, median minimizes L1, mode minimizes L0. A single-number statistic is _going_ to leave things out, so if you must boil thing…

Glad you enjoyed that post so much. It really is a shame that we do such a bad job of teaching students about the inherent subjectivity of descriptive statistics and let students leave their courses with dangerous ideas about the existence of a Holy Grail statistic that will solve all of their problems.

Your followup post (http://www.johnmyleswhite.com/notebook/2013/03/22/using-norm...) is excellent. Thank you!
Post reply on HN