Statistical Mistakes and How to Avoid Them • Hacker News Board

Re: Statistical Mistakes and How to Avoid Them

#51

Post by fela » Thu, Nov 24, 2016, 11:40 AM UTC

"it’s telling you that there’s at most an alpha chance that the difference arose from random chance. In 95 out of 100 parallel universes, your paper found a difference that actually exists. I’d take that bet."

This is wrong. It’s telling you that there’s at most an alpha chance that a difference like that (or more) would have arisen from random chance if the quantities are actually equal. And if the quantities are equal 95 out of 100 parallel universes would not be able to reject the null hypothesis.

Is he saying that he would take the xkcd bet[0] on the frequentist side?

[0] https://xkcd.com/1132/

Re: Statistical Mistakes and How to Avoid Them

#52

Post by throw_away_777 » Thu, Nov 24, 2016, 1:12 PM UTC

lisper wrote: post #32

qwrusz wrote: post #28

Earlier quoted context omitted.

Do you mean probably wrong? Either way I disagree. Both approaches have pros and cons depending on the type of analysis. At the very least, just the presence of competing approaches in the field has pushed statisticians to have more rigour and do way more double checking than they might have, out of fear the other side actively looking to poke holes. It's easy to lie with statistics and the only people who can call s…

> Do you mean probably wrong? Nope. > Both approaches have pros and cons What are the pros of the frequentist approach?

The main advantage of the frequentist approach is that you can do the calculations much more easily. Bayesian statistics is great, but often the calculations are much more difficult because of your distribution of priors. You can make up simplified priors to ease the calculations, but then you run into some of the same problems as frequentist statistics.

Here is a simple example: lets say you flip a coin 10 times and get 8 heads, what is the probability that the coin is not fair? In frequentist statistics you only have to calculate the likelihood of the results for a null hypothesis, and then you use a p-value. While this approach is flawed, at least you can quickly do the calculation and get an approximate answer. In Bayesian statistics you have to specify the prior distribution, and calculate the likelihood of your results under every possible hypothesis. Correctly specifying this prior distribution and calculating the results is quite challenging - especially if you want to use a realistic prior (not just uniform). This is a pretty simple example, you can imagine how much more challenging this becomes in real-world problems. On the other hand, it is true that the frequentist approach doesn't really answer the question asked, so it is misleading (especially if you choose a p-value that isn't specific to the problem). If you choose p-values based on prior knowledge, than the differences between frequentist and bayesian are less extreme.

Re: Statistical Mistakes and How to Avoid Them

#53

Post by throw_away_777 » Thu, Nov 24, 2016, 1:20 PM UTC

amelius wrote: post #49

samps wrote: post #12

Earlier quoted context omitted.

Needless to say, I disagree. It can be straight-up misleading to report means without including a more nuanced view of the distribution. You don't need to use a bunch of fancy statistics, but you do need to consider whether your results could have arisen by random chance. That's not a distraction; it's accurately reporting what you found. Here's one frightening example of spurious performance results in CS: https://w…

> It can be straight-up misleading It is only misleading if the reader doesn't understand statistics. There is, imho, nothing wrong with putting all your focus on the subject matter, and skipping the statistics while being frank about it. Also, if you need statistics to show that your method is better than other methods, then perhaps your method is not really that much better.

If your analysis involves data, you can't skip the statistics. If there is no analysis of data than go ahead and skip the stats all you want. You need to try and determine the uncertainty in your results, both systematic and statistical.

Re: Statistical Mistakes and How to Avoid Them

#54

Post by thanatropism » Thu, Nov 24, 2016, 1:21 PM UTC

srean wrote: post #26

skybrian wrote: post #25

I'm not a statistician but even so I think this article makes assumptions that may not hold up for computer science. The first thing to do is plot your data. If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here. If you're doing benchmarking, another common model is a peak at a minimum value (when everything goes right) and a lon…

> If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here. The key word in there is common . There is an entire industry of statistical techniques that do not require Gaussian assumption or for that matter any parametric assumption. I strongly feel it is time to retire the Gaussian distribution from the space it occupies. Discoveri…

The Gaussian distribution is central in continuous-time models for different reasons:

https://almostsure.wordpress.com/2010/04/13/levys-characteri...

Basically any reasonable* stochastic continuous process is driven by a brownian motion. Also: discontinuous processes are more or less* the sum of a brownian motion and a poisson-type process.

https://en.wikipedia.org/wiki/L%C3%A9vy_process#L.C3.A9vy.E2...

(* Much details about filtrations, Banach spaces yadda yadda omitted)

Re: Statistical Mistakes and How to Avoid Them

#55

Post by platz » Thu, Nov 24, 2016, 2:00 PM UTC

srean wrote: post #42

platz wrote: post #30

Earlier quoted context omitted.

Too cryptic to be helpful, sorry..

Ah I see, my apologies, I assumed Googling those keywords would be sufficient to connect the dots. The main thing that I wanted to convey is that the consequences of CLT does not come for free. It is not remotely as widely applicable as it is made out to be. CLT is also not so narrow that you need IID random variables as is often claimed. Those assumptions can be relaxed substantially. What gets in the way in obtaini…

I wasn't aware of the idea that natural processes might not have a finite variance, or what implications that would have. Thanks for clarifying!

Re: Statistical Mistakes and How to Avoid Them

#56

Post by ska » Thu, Nov 24, 2016, 2:26 PM UTC

throw_away_777 wrote: post #52

lisper wrote: post #32

Earlier quoted context omitted.

> Do you mean probably wrong? Nope. > Both approaches have pros and cons What are the pros of the frequentist approach?

The main advantage of the frequentist approach is that you can do the calculations much more easily. Bayesian statistics is great, but often the calculations are much more difficult because of your distribution of priors. You can make up simplified priors to ease the calculations, but then you run into some of the same problems as frequentist statistics. Here is a simple example: lets say you flip a coin 10 times and…

> Bayesian statistics is great, but often the calculations are much more difficult because of your distribution of priors.

This is often a feature, forcing you to actually look at the complexity head on before you sweep it under a rug.

Re: Statistical Mistakes and How to Avoid Them

#57

Post by ska » Thu, Nov 24, 2016, 2:29 PM UTC

amelius wrote: post #5

I don't like how the article tries to push statistics on the reader. If a CS paper compares a pair of averages, then that gives certain information. If statistics can add to that, and make the results a little more precise, then that is nice. But by no means is it absolutely necessary. And statistics will not give a conclusive result either. I think that authors should use statistics when they see fit, and when it do…

This thinking is part of the problem. While an individual average may just be "a fact", you cannot meaningfully compare two averages without knowing more than their values.

Pretending you can has lead to a lot of muddled thinking.

Re: Statistical Mistakes and How to Avoid Them

#58

Post by lisper » Thu, Nov 24, 2016, 2:42 PM UTC

throw_away_777 wrote: post #52

lisper wrote: post #32

Earlier quoted context omitted.

> Do you mean probably wrong? Nope. > Both approaches have pros and cons What are the pros of the frequentist approach?

The main advantage of the frequentist approach is that you can do the calculations much more easily. Bayesian statistics is great, but often the calculations are much more difficult because of your distribution of priors. You can make up simplified priors to ease the calculations, but then you run into some of the same problems as frequentist statistics. Here is a simple example: lets say you flip a coin 10 times and…

A man was walking down a city street when he saw another man wandering around a lamp post looking at the sidewalk. "What are you doing?" the first man asked. "Looking for my keys," said the second man. "Oh, did you lose them around here?" asked the first man. "No," the second man replied, "but the light is better here."

Re: Statistical Mistakes and How to Avoid Them

#59

Post by nonbel » Thu, Nov 24, 2016, 3:36 PM UTC

lisper wrote: post #21

nonbel wrote: post #18

Earlier quoted context omitted.

>"Statistics alone cannot tell you which of those two models is correct." Statistics can tell you whether a model is consistent with the data. But you need to deduce the null hypothesis from your model rather than use the default "no difference" (of course, sometimes no difference is deduced from a real model, but not often, in that case: great!). In fact, that is the proper use of statistics. I would guess >99.99% o…

> Statistics can tell you whether a model is consistent with the data. Yes, that's true, but it badly misses the point. The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out. Any finite data set is consistent with an infinite number of models, so knowing that a model and the data are consistent tells you absolutely…

>"The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out"

If you know whether your model is consistent with the data, you know whether it is inconsistent... I think you are talking about some other issue than I am.

The point about deducing the null hypothesis from your explanatory model is that the null hypothesis is precise. In that case you will get a strong test of the model, and it will get stronger as more data gets collected. Using a default null and and vague alternative is the exact opposite. (check the Meehl 1967 paper I linked earlier in this thread).

Re: Statistical Mistakes and How to Avoid Them

#60

Post by amelius » Thu, Nov 24, 2016, 3:36 PM UTC

ska wrote: post #57

amelius wrote: post #5

I don't like how the article tries to push statistics on the reader. If a CS paper compares a pair of averages, then that gives certain information. If statistics can add to that, and make the results a little more precise, then that is nice. But by no means is it absolutely necessary. And statistics will not give a conclusive result either. I think that authors should use statistics when they see fit, and when it do…

This thinking is part of the problem. While an individual average may just be "a fact", you cannot meaningfully compare two averages without knowing more than their values. Pretending you can has lead to a lot of muddled thinking.

Assuming distributions that don't correspond to reality can also lead to a lot of muddled thinking.