That's a good article but pretty short. There would be a lot more ground to cover.
Statistical Mistakes and How to Avoid Them
11–20 of 80 posts
Re: Statistical Mistakes and How to Avoid Them
#12I don't like how the article tries to push statistics on the reader. If a CS paper compares a pair of averages, then that gives certain information. If statistics can add to that, and make the results a little more precise, then that is nice. But by no means is it absolutely necessary. And statistics will not give a conclusive result either. I think that authors should use statistics when they see fit, and when it do…
Needless to say, I disagree. It can be straight-up misleading to report means without including a more nuanced view of the distribution. You don't need to use a bunch of fancy statistics, but you do need to consider whether your results could have arisen by random chance. That's not a distraction; it's accurately reporting what you found.
Here's one frightening example of spurious performance results in CS: https://www.cis.upenn.edu/~cis501/papers/producing-wrong-dat...
Re: Statistical Mistakes and How to Avoid Them
#13The t-test assumes a normal distribution which, is rarely true, especially when the number of runs is under 100. A better test is the Mann-Whitney U test which is applicable for a wider category of distributions.
Central limit theorem means there are lots of cases where normal distributions are directly applicable.
The central limit theorem applies to independent variables only. If you are not sure your variables are independent you cannot rely on that assumption.
Re: Statistical Mistakes and How to Avoid Them
#14This is the insight that made statistics "click" for me many years ago: a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance? If those odds are low, then you are justified in concluding that the results probably did not arise by chance, and so there must be some other explanation (usually, but not always, the causal hypothesis you are advancing).…
> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance Well, no, that'd be very interesting but unfortunately what a statistical test really says is the probability of the results you observed (or more extreme) given chance. P(data|model) and not P(model|data).
[deleted]
Re: Statistical Mistakes and How to Avoid Them
#15Earlier quoted context omitted.
It's actually P(data|null-hypothesis).
The model is the null hypothesis here... I don't think the parent comment was wrong w.r.t. this.
No, there's a difference. The model is something like, "This drug attaches itself selectively to cancer cells and kills them." The null hypothesis is, "This drug has no effect." So you conduct a double-blind study, measure the effect of the drug on cancer cells, collect some data and compute that P(data|null-hypothesis) is 1%. It it not the case that there is a 99% chance that your model is correct and that the drug does in fact attach itself to cancer cells and kill them. Because there are other possible models, e.g.: the drug redirects your Chi and channels it into your chakras which kills the cancer cells. Statistics alone cannot tell you which of those two models is correct.
Re: Statistical Mistakes and How to Avoid Them
#16Earlier quoted context omitted.
Central limit theorem means there are lots of cases where normal distributions are directly applicable.
The central limit theorem applies to independent variables only. If you are not sure your variables are independent you cannot rely on that assumption.
Good to know!
Re: Statistical Mistakes and How to Avoid Them
#17Earlier quoted context omitted.
The model is the null hypothesis here... I don't think the parent comment was wrong w.r.t. this.
No, there's a difference. The model is something like, "This drug attaches itself selectively to cancer cells and kills them." The null hypothesis is, "This drug has no effect." So you conduct a double-blind study, measure the effect of the drug on cancer cells, collect some data and compute that P(data|null-hypothesis) is 1%. It it not the case that there is a 99% chance that your model is correct and that the drug…
I think this is just an argument over semantics, the "null hypothesis" is a perfectly valid model using the definition of the statistics community [0] (i.e. a collection of probability distributions over some sample space).
Re: Statistical Mistakes and How to Avoid Them
#18Earlier quoted context omitted.
The model is the null hypothesis here... I don't think the parent comment was wrong w.r.t. this.
No, there's a difference. The model is something like, "This drug attaches itself selectively to cancer cells and kills them." The null hypothesis is, "This drug has no effect." So you conduct a double-blind study, measure the effect of the drug on cancer cells, collect some data and compute that P(data|null-hypothesis) is 1%. It it not the case that there is a 99% chance that your model is correct and that the drug…
>"Statistics alone cannot tell you which of those two models is correct."
Statistics can tell you whether a model is consistent with the data. But you need to deduce the null hypothesis from your model rather than use the default "no difference" (of course, sometimes no difference is deduced from a real model, but not often, in that case: great!).
In fact, that is the proper use of statistics. I would guess >99.99% of current usage is incorrect (ie pseudoscience) and amounts to a waste of time at best. The usual usage turns scientific reasoning on its head, and has lead to a (literally for most people) unbelievable amount of trouble.
This was pointed out most aptly by Paul Meehl long, long ago: http://www.fisme.science.uu.nl/staff/christianb/downloads/me...
Re: Statistical Mistakes and How to Avoid Them
#19Earlier quoted context omitted.
Central limit theorem means there are lots of cases where normal distributions are directly applicable.
The central limit theorem applies to independent variables only. If you are not sure your variables are independent you cannot rely on that assumption.
That's not 100% true. There are lots of different theorems that are "central limit theorems", and that work across different cases.
You can have CLT's with non-iid variables (either the aren't identically distributed or aren't independent). The math just becomes much harder, and you have to assume specific dependence structures.
For example https://en.wikipedia.org/wiki/Martingale_central_limit_theor...
Re: Statistical Mistakes and How to Avoid Them
#20Earlier quoted context omitted.
The central limit theorem applies to independent variables only. If you are not sure your variables are independent you cannot rely on that assumption.
That's not 100% true. There are lots of different theorems that are "central limit theorems", and that work across different cases. You can have CLT's with non-iid variables (either the aren't identically distributed or aren't independent). The math just becomes much harder, and you have to assume specific dependence structures. For example https://en.wikipedia.org/wiki/Martingale_central_limit_theor...
Thanks for the reference - can you provide an example where one would use the Martingale CLT, if you are aware of any ?