Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistical Mistakes and How to Avoid Them

cs.cornell.edu

Post reply on HN
Print view View original

21–30 of 80 posts

Re: Statistical Mistakes and How to Avoid Them

#21

Post by lisper »

nonbel wrote: post #18
lisper wrote: post #15

Earlier quoted context omitted.

No, there's a difference. The model is something like, "This drug attaches itself selectively to cancer cells and kills them." The null hypothesis is, "This drug has no effect." So you conduct a double-blind study, measure the effect of the drug on cancer cells, collect some data and compute that P(data|null-hypothesis) is 1%. It it not the case that there is a 99% chance that your model is correct and that the drug…

>"Statistics alone cannot tell you which of those two models is correct." Statistics can tell you whether a model is consistent with the data. But you need to deduce the null hypothesis from your model rather than use the default "no difference" (of course, sometimes no difference is deduced from a real model, but not often, in that case: great!). In fact, that is the proper use of statistics. I would guess >99.99% o…

> Statistics can tell you whether a model is consistent with the data.

Yes, that's true, but it badly misses the point. The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out. Any finite data set is consistent with an infinite number of models, so knowing that a model and the data are consistent tells you absolutely nothing about whether or not that model has any relationship with reality (which, at the risk of stating the obvious, is what science actually cares about). This is the reason that rejecting the null hypothesis is considered a positive result.

Re: Statistical Mistakes and How to Avoid Them

#22

Post by qwrusz »

lisper wrote: post #6
stdbrouw wrote: post #4

Earlier quoted context omitted.

> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance Well, no, that'd be very interesting but unfortunately what a statistical test really says is the probability of the results you observed (or more extreme) given chance. P(data|model) and not P(model|data).

It's actually P(data|null-hypothesis).

How about you guys are both right, sort of.

There are both Bayesian and Frequentist approaches in statistics!

They represent very different methods to statistics, but they are also quite similar. My apologies, I couldn't find one link that gave a good description of Bayesian vs. Frequentist. Here are a couple links to get started:

https://xkcd.com/1132/

http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bay...

If anyone comes across a link that describes Bayesian and Frequentist clearly, please share it if you don't mind.

Re: Statistical Mistakes and How to Avoid Them

#23

Post by lisper »

fluxion wrote: post #17
lisper wrote: post #15

Earlier quoted context omitted.

No, there's a difference. The model is something like, "This drug attaches itself selectively to cancer cells and kills them." The null hypothesis is, "This drug has no effect." So you conduct a double-blind study, measure the effect of the drug on cancer cells, collect some data and compute that P(data|null-hypothesis) is 1%. It it not the case that there is a 99% chance that your model is correct and that the drug…

I think this is just an argument over semantics, the "null hypothesis" is a perfectly valid model using the definition of the statistics community [0] (i.e. a collection of probability distributions over some sample space). [0] https://en.wikipedia.org/wiki/Statistical_model

The word "model" is being used here in two mutually incompatible ways. Yes, the null hypothesis is a model, but it is not an explanatory model. A scientific hypothesis has to meet two tests to be considered a valid theory. It has to be consistent with the data, and it has to have explanatory power. The theory that cancer drugs work by aligning a patient's chi with their chakras is rejected not because it is inconsistent with the data (it's not) but because it lacks the explanatory power of alternative theories based on molecular biology.

The null hypothesis never has explanatory power. The null hypothesis is always a statement of the form, "The explanatory hypothesis under test is wrong for some unknown reason." This is why rejecting the null hypothesis, i.e. showing that the data are (with high probability) inconsistent with the null hypothesis, is considered a positive result.

Re: Statistical Mistakes and How to Avoid Them

#24

Post by lisper »

qwrusz wrote: post #22
lisper wrote: post #6

Earlier quoted context omitted.

It's actually P(data|null-hypothesis).

How about you guys are both right, sort of. There are both Bayesian and Frequentist approaches in statistics! They represent very different methods to statistics, but they are also quite similar. My apologies, I couldn't find one link that gave a good description of Bayesian vs. Frequentist. Here are a couple links to get started: https://xkcd.com/1132/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bay...…

That cartoon is actually a pretty good illustration of why frequentists are wrong.

Re: Statistical Mistakes and How to Avoid Them

#25

Post by skybrian »

I'm not a statistician but even so I think this article makes assumptions that may not hold up for computer science. The first thing to do is plot your data. If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here.

If you're doing benchmarking, another common model is a peak at a minimum value (when everything goes right) and a long tail, due to various events like cache misses that always slow things down, but don't happen in every test run.

On a system with multiple programs running (a typical desktop), taking the mean is meaningless - this just adds noise due to activity unrelated to your program. You'd be better off taking the minimum, which with enough test runs should capture all the events that happen every time and none of the events that don't.

The median or 95% percentile might also be useful if you're investigating events that don't happen every time. But if you want to know about cold start performance (for example), maybe the best thing to do would be to flush your caches before every test run, so the events you're interested in are events that happen every time.

Re: Statistical Mistakes and How to Avoid Them

#26

Post by srean »

skybrian wrote: post #25

I'm not a statistician but even so I think this article makes assumptions that may not hold up for computer science. The first thing to do is plot your data. If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here. If you're doing benchmarking, another common model is a peak at a minimum value (when everything goes right) and a lon…

> If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here.

The key word in there is common. There is an entire industry of statistical techniques that do not require Gaussian assumption or for that matter any parametric assumption.

I strongly feel it is time to retire the Gaussian distribution from the space it occupies. Discovering and studying Gaussian distribution and the bog standard central limit theorem should be considered one of mankind's crowning achievements. They deserve to be put on a pedestal to appreciate their elegance, but when rubber meets the road one has to open ones mind to look beyond. Appearance of the Gaussian distribution is rarely as normal as many expect/claim it to be (I blame the stats education machinery for this), nor was it invented by Gauss. In fact Gauss used it as a post-hoc justification for backing the least-squares method. His original motivation for least squares was simplicity and convenience, not the normal distribution or CLT or for that matter the Gauss-Markov theorem.

Re: Statistical Mistakes and How to Avoid Them

#27

Post by srean »

ekianjo wrote: post #20
ronald_raygun wrote: post #19

Earlier quoted context omitted.

That's not 100% true. There are lots of different theorems that are "central limit theorems", and that work across different cases. You can have CLT's with non-iid variables (either the aren't identically distributed or aren't independent). The math just becomes much harder, and you have to assume specific dependence structures. For example https://en.wikipedia.org/wiki/Martingale_central_limit_theor...

Thanks for the reference - can you provide an example where one would use the Martingale CLT, if you are aware of any ?

Look for any estimation or inference problems in the context of stochastic processes. For a simpler example you can take a look at sequential test of hypothesis. It is quite ubiquitous, but not always called out by its name.

Re: Statistical Mistakes and How to Avoid Them

#28

Post by qwrusz »

lisper wrote: post #24
qwrusz wrote: post #22

Earlier quoted context omitted.

How about you guys are both right, sort of. There are both Bayesian and Frequentist approaches in statistics! They represent very different methods to statistics, but they are also quite similar. My apologies, I couldn't find one link that gave a good description of Bayesian vs. Frequentist. Here are a couple links to get started: https://xkcd.com/1132/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bay...…

That cartoon is actually a pretty good illustration of why frequentists are wrong.

Do you mean probably wrong?

Either way I disagree. Both approaches have pros and cons depending on the type of analysis.

At the very least, just the presence of competing approaches in the field has pushed statisticians to have more rigour and do way more double checking than they might have, out of fear the other side actively looking to poke holes. It's easy to lie with statistics and the only people who can call statisticians out on their BS is other statisticians...Need more of this.

Re: Statistical Mistakes and How to Avoid Them

#29

Post by srean »

platz wrote: post #9
frozenport wrote: post #7

The t-test assumes a normal distribution which, is rarely true, especially when the number of runs is under 100. A better test is the Mann-Whitney U test which is applicable for a wider category of distributions.

Central limit theorem means there are lots of cases where normal distributions are directly applicable.

Someone ought to formulate the Godwin's law analogue for CLT.

Hint 1: what happens when variance does not exist or is too large to be considered finite for any practical purposes.

Hint 2: Levy processes, stable distributions

Re: Statistical Mistakes and How to Avoid Them

#30

Post by platz »

srean wrote: post #29
platz wrote: post #9

Earlier quoted context omitted.

Central limit theorem means there are lots of cases where normal distributions are directly applicable.

Someone ought to formulate the Godwin's law analogue for CLT. Hint 1: what happens when variance does not exist or is too large to be considered finite for any practical purposes. Hint 2: Levy processes, stable distributions

Too cryptic to be helpful, sorry..
Post reply on HN