Statistical Mistakes and How to Avoid Them
cs.cornell.edu
Statistical Mistakes and How to Avoid Them
1–10 of 80 posts
Re: Statistical Mistakes and How to Avoid Them
#2This is the insight that made statistics "click" for me many years ago: a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance? If those odds are low, then you are justified in concluding that the results probably did not arise by chance, and so there must be some other explanation (usually, but not always, the causal hypothesis you are advancing).
Once consequence of this is that it is crucial that you advance your hypothesis before you collect (or at least look at) the data because the odds of something arising by chance change depending on whether you predict or postdict the results. Also, the more data you have, the more likely you are to find something in there that looks like a signal but is in fact just a coincidence. Many a day-trading fortune has been lost to this one mistake.
Re: Statistical Mistakes and How to Avoid Them
#3[deleted]
Re: Statistical Mistakes and How to Avoid Them
#4This is the insight that made statistics "click" for me many years ago: a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance? If those odds are low, then you are justified in concluding that the results probably did not arise by chance, and so there must be some other explanation (usually, but not always, the causal hypothesis you are advancing).…
> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance
Well, no, that'd be very interesting but unfortunately what a statistical test really says is the probability of the results you observed (or more extreme) given chance. P(data|model) and not P(model|data).
Re: Statistical Mistakes and How to Avoid Them
#5I don't like how the article tries to push statistics on the reader. If a CS paper compares a pair of averages, then that gives certain information. If statistics can add to that, and make the results a little more precise, then that is nice. But by no means is it absolutely necessary. And statistics will not give a conclusive result either.
I think that authors should use statistics when they see fit, and when it does not distract too much from the original subject of the paper.
Re: Statistical Mistakes and How to Avoid Them
#6This is the insight that made statistics "click" for me many years ago: a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance? If those odds are low, then you are justified in concluding that the results probably did not arise by chance, and so there must be some other explanation (usually, but not always, the causal hypothesis you are advancing).…
> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance Well, no, that'd be very interesting but unfortunately what a statistical test really says is the probability of the results you observed (or more extreme) given chance. P(data|model) and not P(model|data).
It's actually P(data|null-hypothesis).
Re: Statistical Mistakes and How to Avoid Them
#7The t-test assumes a normal distribution which, is rarely true, especially when the number of runs is under 100. A better test is the Mann-Whitney U test which is applicable for a wider category of distributions.
Re: Statistical Mistakes and How to Avoid Them
#8Earlier quoted context omitted.
> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance Well, no, that'd be very interesting but unfortunately what a statistical test really says is the probability of the results you observed (or more extreme) given chance. P(data|model) and not P(model|data).
It's actually P(data|null-hypothesis).
The model is the null hypothesis here... I don't think the parent comment was wrong w.r.t. this.
Re: Statistical Mistakes and How to Avoid Them
#9The t-test assumes a normal distribution which, is rarely true, especially when the number of runs is under 100. A better test is the Mann-Whitney U test which is applicable for a wider category of distributions.
Central limit theorem means there are lots of cases where normal distributions are directly applicable.
Re: Statistical Mistakes and How to Avoid Them
#10I don't like how the article tries to push statistics on the reader. If a CS paper compares a pair of averages, then that gives certain information. If statistics can add to that, and make the results a little more precise, then that is nice. But by no means is it absolutely necessary. And statistics will not give a conclusive result either. I think that authors should use statistics when they see fit, and when it do…
IMO it's better to just plot the two distributions that you want to compare so that people can eyeball the difference (or run the statistical tests themselves should they wish to do so). For one thing not all distributions are Gaussian. And the t-test only answers one specific question (i.e., with one given p-value). Then there's people who misinterpret the result of the t-test. Or people who mess with the p-value till they get what they need.