Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistical Mistakes and How to Avoid Them

cs.cornell.edu

Post reply on HN
Print view View original

31–40 of 80 posts

Re: Statistical Mistakes and How to Avoid Them

#31

Post by aristus »

qwrusz wrote: post #22
lisper wrote: post #6

Earlier quoted context omitted.

It's actually P(data|null-hypothesis).

How about you guys are both right, sort of. There are both Bayesian and Frequentist approaches in statistics! They represent very different methods to statistics, but they are also quite similar. My apologies, I couldn't find one link that gave a good description of Bayesian vs. Frequentist. Here are a couple links to get started: https://xkcd.com/1132/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bay...…

Frequentists count their chickens after they've hatched. Bayesians bet the chickens that they will.

Re: Statistical Mistakes and How to Avoid Them

#32

Post by lisper »

qwrusz wrote: post #28
lisper wrote: post #24

Earlier quoted context omitted.

That cartoon is actually a pretty good illustration of why frequentists are wrong.

Do you mean probably wrong? Either way I disagree. Both approaches have pros and cons depending on the type of analysis. At the very least, just the presence of competing approaches in the field has pushed statisticians to have more rigour and do way more double checking than they might have, out of fear the other side actively looking to poke holes. It's easy to lie with statistics and the only people who can call s…

> Do you mean probably wrong?

Nope.

> Both approaches have pros and cons

What are the pros of the frequentist approach?

Re: Statistical Mistakes and How to Avoid Them

#33

Post by kgwgk »

stdbrouw wrote: post #4
lisper wrote: post #2

This is the insight that made statistics "click" for me many years ago: a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance? If those odds are low, then you are justified in concluding that the results probably did not arise by chance, and so there must be some other explanation (usually, but not always, the causal hypothesis you are advancing).…

> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance Well, no, that'd be very interesting but unfortunately what a statistical test really says is the probability of the results you observed (or more extreme) given chance. P(data|model) and not P(model|data).

> what are the odds that the results you observed could have arisen by chance?

> the probability of the results you observed (or more extreme) given chance

I think he said the same thing as you. He didn't talk about "the odds that the results you observed have arisen by chance" but the odds that the results you have observed (the or more extreme is implicit) could have arisen by chance (i.e. the probability of happening by chance alone, given that only chance was involved).

Re: Statistical Mistakes and How to Avoid Them

#34

Post by ACow_Adonis »

lisper wrote: post #21
nonbel wrote: post #18

Earlier quoted context omitted.

>"Statistics alone cannot tell you which of those two models is correct." Statistics can tell you whether a model is consistent with the data. But you need to deduce the null hypothesis from your model rather than use the default "no difference" (of course, sometimes no difference is deduced from a real model, but not often, in that case: great!). In fact, that is the proper use of statistics. I would guess >99.99% o…

> Statistics can tell you whether a model is consistent with the data. Yes, that's true, but it badly misses the point. The power of statistics is to tell you when a model (the null hypothesis) is (most likely) inconsistent with the data so that you can confidently rule it out. Any finite data set is consistent with an infinite number of models, so knowing that a model and the data are consistent tells you absolutely…

Except such a data set is also inconsistent with an infinite number of models, so ruling one out via rejecting a null also provides practically no information value and moves us no closer to understanding. /devils advocate

In practical terms, we're not interested in true models, but useful ones, so the description of a model's consistency with observed data is often the more useful metric in practice than rejecting nulls :/ especially in applications where you can't set up repeated experiments.

OK, I realise its more nuanced than that too, but given how many papers and practitioners seem incapable of understanding that evidence against the null it's not explicit evidence for an arbitrary alternative, practically and consequentially I don't think that's how we should be working...

Re: Statistical Mistakes and How to Avoid Them

#35

Post by danieltillett »

lisper wrote: post #32
qwrusz wrote: post #28

Earlier quoted context omitted.

Do you mean probably wrong? Either way I disagree. Both approaches have pros and cons depending on the type of analysis. At the very least, just the presence of competing approaches in the field has pushed statisticians to have more rigour and do way more double checking than they might have, out of fear the other side actively looking to poke holes. It's easy to lie with statistics and the only people who can call s…

> Do you mean probably wrong? Nope. > Both approaches have pros and cons What are the pros of the frequentist approach?

Simple maths that can be done without a computer. Useful in the past - not so useful today.

Re: Statistical Mistakes and How to Avoid Them

#36

Post by rcthompson »

amelius wrote: post #5

I don't like how the article tries to push statistics on the reader. If a CS paper compares a pair of averages, then that gives certain information. If statistics can add to that, and make the results a little more precise, then that is nice. But by no means is it absolutely necessary. And statistics will not give a conclusive result either. I think that authors should use statistics when they see fit, and when it do…

Any time you compare two averages, you are doing statistics, whether or not you report the result in statistical terms. It's not something optional that you can "add on" to provide extra information. If you don't provide some measure of significance, I'm not going to trust that your result has any chance of being real. At best, you don't really know whether the result is real because you ignored the statistics; at worst, you ran the statistics and know it's not real, and you're hoping I won't notice.

Re: Statistical Mistakes and How to Avoid Them

#37

Post by rcthompson »

frozenport wrote: post #7

The t-test assumes a normal distribution which, is rarely true, especially when the number of runs is under 100. A better test is the Mann-Whitney U test which is applicable for a wider category of distributions.

I think the t-test is conceptually easier to understand, which is important since the target audience for this article is people who know next to nothing about statistics.

The t-test might not be the best test for every situation, but if the alternative is no test at all, I'll take it.

Re: Statistical Mistakes and How to Avoid Them

#38

Post by rspeer »

lisper wrote: post #32
qwrusz wrote: post #28

Earlier quoted context omitted.

Do you mean probably wrong? Either way I disagree. Both approaches have pros and cons depending on the type of analysis. At the very least, just the presence of competing approaches in the field has pushed statisticians to have more rigour and do way more double checking than they might have, out of fear the other side actively looking to poke holes. It's easy to lie with statistics and the only people who can call s…

> Do you mean probably wrong? Nope. > Both approaches have pros and cons What are the pros of the frequentist approach?

You can justify that you did everything "by the book" and you don't have to make up priors.

Now. You might read this as me saying that Bayesian statistics is all made up, and that's not what I'm saying. I'm saying that if you go Bayesian all the way, and your result is even remotely controversial, someone could easily challenge you by saying "this result would have been different with different priors, and why won't they just come out and say p When you do frequentist statistics, the equivalent of priors were, instead, made up for you by long scientific tradition. They're not very good priors, as the relevant xkcd illustrates. But at least it's not you making them up.

In a more complex model, there's also the thing where you can't calculate the result exactly, you have to approximate it with Markov Chain Monte Carlo or something, leading to another way to doubt your results (did the MCMC converge correctly?)

So instead you do frequentist statistics. You use your favorite stats package and it spits out a nice comforting p-value that will satisfy the reviewers. You have tons of guidance about how to do things. It's not a great thing, but it is a definite advantage of using frequentist statistics.

Re: Statistical Mistakes and How to Avoid Them

#39

Post by stdbrouw »

lisper wrote: post #6
stdbrouw wrote: post #4

Earlier quoted context omitted.

> a statistical test answers one central question: what are the odds that the results you observed could have arisen by chance Well, no, that'd be very interesting but unfortunately what a statistical test really says is the probability of the results you observed (or more extreme) given chance. P(data|model) and not P(model|data).

It's actually P(data|null-hypothesis).

Yup, and the null hypothesis is a reduced model. Different words, same thing.

Re: Statistical Mistakes and How to Avoid Them

#40

Post by stdbrouw »

qwrusz wrote: post #22
lisper wrote: post #6

Earlier quoted context omitted.

It's actually P(data|null-hypothesis).

How about you guys are both right, sort of. There are both Bayesian and Frequentist approaches in statistics! They represent very different methods to statistics, but they are also quite similar. My apologies, I couldn't find one link that gave a good description of Bayesian vs. Frequentist. Here are a couple links to get started: https://xkcd.com/1132/ http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bay...…

There are some cases when the difference between bayesian and frequentist statistics is just a matter of interpretation, but not in this case: the mathematics for a bayesian statistical test are different from the mathematics of a frequentist statistical test and they will produce different results, because the former accounts for prior probabilities and the latter does not. The article talks about frequentist tests, as does almost everyone else who mentions statistical tests without further specification.
Post reply on HN