Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistical Mistakes and How to Avoid Them

cs.cornell.edu

Post reply on HN
Print view View original

61–70 of 80 posts

Re: Statistical Mistakes and How to Avoid Them

#61

Post by rspeer »

lisper wrote: post #46
rspeer wrote: post #38

Earlier quoted context omitted.

You can justify that you did everything "by the book" and you don't have to make up priors. Now. You might read this as me saying that Bayesian statistics is all made up, and that's not what I'm saying. I'm saying that if you go Bayesian all the way, and your result is even remotely controversial, someone could easily challenge you by saying "this result would have been different with different priors, and why won't…

In other words: the advantage of frequentism is that it gives you better plausible deniability when you get the wrong answer. Did I get that right?

No, and I don't know where you're getting your certainty that the answer is wrong from.

How about this: find me a published scientific paper with only Bayesian results in it, no frequentist statistics at all. Argue for its superiority all you want, but I don't think anyone does it. It would be seen as a stunt.

Re: Statistical Mistakes and How to Avoid Them

#62

Post by ska »

amelius wrote: post #60
ska wrote: post #57

Earlier quoted context omitted.

This thinking is part of the problem. While an individual average may just be "a fact", you cannot meaningfully compare two averages without knowing more than their values. Pretending you can has lead to a lot of muddled thinking.

Assuming distributions that don't correspond to reality can also lead to a lot of muddled thinking.

That's true, but though related, a separate issue.

For example. I compute a arithmetic mean of two data sets A,B yielding means a, b.

I can tell you what the difference |a-b| is without any other information, but I simply can't tell you if it is significant or not.

This has nothing to do with distribution. Given the variances, I can tell you something about the significance (at least in some senses). But without knowing the distribution I can't tell you at all how to interpret the variance.

The point is, as soon as you compute that mean, you are doing statistics. If you do it carefully, you will be able to define what the numbers mean, and what they do not. The fact that many people don't do it well does not change this.

There is no possibility of improving the situation by ignoring how the numbers were arrived at and what that actually means. Sometimes the best thing to come out of it is that you are simply calculating the wrong thing for what you want to learn.

Re: Statistical Mistakes and How to Avoid Them

#64

Post by willhslade »

srean wrote: post #42
platz wrote: post #30

Earlier quoted context omitted.

Too cryptic to be helpful, sorry..

Ah I see, my apologies, I assumed Googling those keywords would be sufficient to connect the dots. The main thing that I wanted to convey is that the consequences of CLT does not come for free. It is not remotely as widely applicable as it is made out to be. CLT is also not so narrow that you need IID random variables as is often claimed. Those assumptions can be relaxed substantially. What gets in the way in obtaini…

I've seen variations of this comment in multiple threads, and have to ask: is there a good paper or textbook that spells this out?

Re: Statistical Mistakes and How to Avoid Them

#65

Post by imh »

wodenokoto wrote: post #63

Is there a good resource to learn the underpinnings of P values and T-tests? I feel like everybody says these are important, show a formula and then arguments ensue about what p=0.95 means, and nobody seems to know this.

I think any intro stats book should do the trick. As far as I know, the material in a first stats course is pretty homogeneous. I'm not a biostatistician, but I happen to like this book [0] for introductory stuff. Amazon says you can get it used for $26.

[0] https://www.amazon.com/Principles-Biostatistics-CD-ROM-Marce...

Re: Statistical Mistakes and How to Avoid Them

#66

Post by imh »

amelius wrote: post #60
ska wrote: post #57

Earlier quoted context omitted.

This thinking is part of the problem. While an individual average may just be "a fact", you cannot meaningfully compare two averages without knowing more than their values. Pretending you can has lead to a lot of muddled thinking.

Assuming distributions that don't correspond to reality can also lead to a lot of muddled thinking.

You can do quite a bit without making strong distributional assumptions. There are nonparametric tests for all sorts of things (and the basic ones are covered in good first intro to stats books too).

Re: Statistical Mistakes and How to Avoid Them

#67

Post by throw_away_777 »

lisper wrote: post #58
throw_away_777 wrote: post #52

Earlier quoted context omitted.

The main advantage of the frequentist approach is that you can do the calculations much more easily. Bayesian statistics is great, but often the calculations are much more difficult because of your distribution of priors. You can make up simplified priors to ease the calculations, but then you run into some of the same problems as frequentist statistics. Here is a simple example: lets say you flip a coin 10 times and…

A man was walking down a city street when he saw another man wandering around a lamp post looking at the sidewalk. "What are you doing?" the first man asked. "Looking for my keys," said the second man. "Oh, did you lose them around here?" asked the first man. "No," the second man replied, "but the light is better here."

Often times getting close to an answer that is approximately correct is better than trying to find the perfect solution. Most real world problems, especially in analyzing data, don't have perfect answers. For example approximations are made all the time in physics, because without these approximations the calculations can't be done. Knowing when to make approximations and what approximations to make is an essential skill for analyzing data.

Re: Statistical Mistakes and How to Avoid Them

#68

Post by lisper »

rspeer wrote: post #61
lisper wrote: post #46

Earlier quoted context omitted.

In other words: the advantage of frequentism is that it gives you better plausible deniability when you get the wrong answer. Did I get that right?

No, and I don't know where you're getting your certainty that the answer is wrong from. How about this: find me a published scientific paper with only Bayesian results in it, no frequentist statistics at all. Argue for its superiority all you want, but I don't think anyone does it. It would be seen as a stunt.

Don't know about a paper, but here's a book:

https://www.amazon.com/Proving-History-Bayess-Theorem-Histor...

Re: Statistical Mistakes and How to Avoid Them

#69

Post by lisper »

nonbel wrote: post #59
lisper wrote: post #21

Earlier quoted context omitted.

> Statistics can tell you whether a model is consistent with the data. Yes, that's true, but it badly misses the point. The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out. Any finite data set is consistent with an infinite number of models, so knowing that a model and the data are consistent tells you absolutely…

>"The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out" If you know whether your model is consistent with the data, you know whether it is inconsistent... I think you are talking about some other issue than I am. The point about deducing the null hypothesis from your explanatory model is that the null hypothesis i…

> If you know whether your model is consistent with the data, you know whether it is inconsistent...

No, that's not true. And in fact I got it wrong earlier when I agreed with you that statistics can tell you when data is consistent with a model. They can't. At best they can tell you whether the data are not inconsistent. That sounds like the same thing, but it isn't. It's like the distinction between "not guilty" and "actually innocent." At best (or at worst depending on how you look at it) a statistical test can tell you, "This theory cannot be confidently ruled out on the basis of that data under the following assumptions..."

> I think you are talking about some other issue than I am.

That is quite possible.

[UPDATE:] BTW, I read the Meehl paper, and I completely agree with what he says. So you and I may be in "violent agreement" here.

Re: Statistical Mistakes and How to Avoid Them

#70

Post by GFK_of_xmaspast »

srean wrote: post #26
skybrian wrote: post #25

I'm not a statistician but even so I think this article makes assumptions that may not hold up for computer science. The first thing to do is plot your data. If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here. If you're doing benchmarking, another common model is a peak at a minimum value (when everything goes right) and a lon…

> If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here. The key word in there is common . There is an entire industry of statistical techniques that do not require Gaussian assumption or for that matter any parametric assumption. I strongly feel it is time to retire the Gaussian distribution from the space it occupies. Discoveri…

I am a huge fan of nonparametric statistics, but lots of non-Gaussian distributions are often 'good enough' for normality assumptions to be ok.
Post reply on HN