Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

fivethirtyeight.com

Post reply on HN
Print view View original

21–30 of 130 posts

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#21

Post by amluto »

If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings?

On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished.

Webster's 1913 dictionary says:

> Deserving to be considered; important; momentous; as, a significant event.

Statisticians don't use "significant" to mean important at all -- they use it to mean "I could detect it". This is bad when someone publishes a paper saying "I found that some drug significantly reduces such-and-such" -- this could just mean that they did a HUGE study and found that the drug reliably had some completely unimportant effect. It's much worse when it's negated, though. Think about all the headlines that say that some treatment "did not have a significant effect". This is basically meaningless. I could do a study finding that exercise has no significant effect on fitness, for example, by making the study small enough.

A good friend of mine suggested that statisticians replace "significant" with "discernible". So next time someone does a small study, they might find that "eating fat had no discernible effect on weight gain", and perhaps readers would then ask the obvious question, which is "how hard did you look?".

This would also help people doing very good research make less wishy-washy conclusions. For example, suppose that "vaccines have no discernible effect on autism rates". This is probably true in a number of studies, but it's the wrong analysis. If researchers who did these studies had to state the conclusions in a silly manner like that, maybe they'd find a more useful analysis to do.

Hint: doing big studies just so you can fail to find an effect is nonsensical. Instead, do big studies so you can put a tight upper bound on the effect. Don't tell me that vaccines don't have a significant (or discernible) effect on autism. Tell me that, with 99.9% confidence, you have ruled out the possibility that vaccines have caused more than ten autism cases in the entire history of vaccines, and that, most likely, they've caused no cases whatsoever (or whatever the right numbers are).

Edit: fixed an insignificant typo.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#22

Post by minimaxir »

eanzenberg wrote: post #15
carbocation wrote: post #10

Earlier quoted context omitted.

> 5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p This would still end up leading to misuse of P-values. Let's say you're doing a genome-wide association study on several hundred thousand SNPs. The traditional threshold is 5e-8 (0.05 / 1,000,000 effective tests). So using 3e-7 for the threshold for "discovery", you'd count many things as discovery that shouldn't be so. On the other hand, let's sa…

Multiple comparisons? 1 million independent tests? Hello? >Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be. >Let's go 5 sigma (p ^Means you should start a new trial with n > 20 given the same placebo/drug split.

In the scientific world, it is economically infeasable (cost/time) for 1 million tests for a given hypothesis. Even n > 20 can be difficult for certain studies. Bootstrapping the results to simulate 1 million trials won't fix the aforementioned issue either.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#23

Post by irremediable »

darawk wrote: post #8

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

Nope! It's "the probability of chance* leading to something at least as extreme as your results". They're related, but they're not the same.

* Ok, a chosen null hypothesis, often chosen to be something like "chance".

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#24

Post by capnrefsmmat »

darawk wrote: post #8

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

You have to be very specific about "probability that they occurred by chance." If p = 0.05, you can't say "there's a 95% chance this result is real and a 5% chance it's just a fluke." You can say "if chance is the only thing operating, we'd see a result like this only 5% of the time."

In conditional probability notation, it's the difference between P(result | it's just chance) and P(it's just chance | result).

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#25

Post by capnrefsmmat »

eanzenberg wrote: post #5

5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at.

That's fine, but then your power is low, and nearly every result you get will be an exaggeration. The more stringent your p value threshold, the more dramatic your results must be to be significant; if your sample size isn't adequate, you'll only get significance if you overestimate the effect.

This is an enormously common problem even with current p value thresholds. It's part of the reason why you see dramatic "A causes B!" results followed by replications saying "well, only a little."

http://www.statisticsdonewrong.com/regression.html#truth-inf...

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#26

Post by jpeterson »

darawk wrote: post #8

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

Is the p-value really not the probability of your results being due to chance?

No, it's the probability of a particular observation, given that we assume the result is due to chance. This sounds similar, but the difference is that it doesn't say anything about the probability of your result outside the context of the study's hypotheses.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#27

Post by evolsb »

amluto wrote: post #21

If they held the same meeting 20 times, would they reach the same conclusion in 19 of those meetings? On a more serious note, I think that the use of the word "significant" to mean "the effect is reasonably likely to exist by some standard" should be abolished. Webster's 1913 dictionary says: > Deserving to be considered; important; momentous; as, a significant event. Statisticians don't use "significant" to mean imp…

There is a significant (pun) difference between statistically significant and economically significant. The conclusion in the drug paper conflates the two. In finance, we could find plenty of statistically significant results (e.g. small cap stocks outperform large cap stocks on Fridays, with a small p-value if you like), but most results were not economically significant--they were not usable for a trading system because they were too small to overcome real-world costs. In short, they weren't meaningful in a real world sense, even though the result was detectable statistically.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#28

Post by thearn4 »

tokenadult wrote: post #16

The article submitted here leads to the American Statistical Association statement on the meaning of p values,[1] the first such methodological statement ever formally issued by the association. It's free to read and download. The statement summarizes into these main points, with further explanation in the text of the statement. "What is a p-value? "Informally, a p-value is the probability under a specified statistic…

I remember the feeling after my first undergraduate course in statistics essentially being that we stated these principles, then spent the remaining weeks essentially invalidating them without offering any real alternatives. My professor may have been more careful than I remember, but the subtleness was lost on me at the time if that was the case.

The testing of statistical hypotheses always seemed like an odd area of the mathematical sciences to me, even after later taking a graduate mathematical statistics sequence. Like an academic squabble between giants in the field of frequentist inference (Fisher vs. Neyman and Pearson) that ended suddenly without resolution, with the scientific community decided to sloppily merge the two positions for the purposes of publication and forge onward.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#29

Post by [deleted] »

darawk wrote: post #8

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

[deleted]

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#30

Post by zamalek »

btilly wrote: post #6

I agree as well! Here is what probability theory teaches us. The proper role of data is to adjust our prior beliefs about probabilities to posterior beliefs through Bayes' theorem. The challenge is how to best communicate this result to people who may have had a wide range of prior beliefs. p-values capture a degree of surprise in the result. Naively, a surprising result should catch our attention and cause us to ret…

The way I see p-values is a mathematical way to share how certain you are that something is in the realm of reality (not truth). It is not a formal conclusion of the results, and is nothing more than a "spoiler" indicating what you might also conclude.
Post reply on HN