Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

fivethirtyeight.com

Post reply on HN
Print view View original

11–20 of 130 posts

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#11

Post by chimeracoder »

darawk wrote: post #8

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

> I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false?

It is the probability of achieving a result at least as far from the hypothesized value exclusively due to random variation that is uncorrelated with the explanatory variable(s) at hand [and subject to a number of other assumptions].

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#12

Post by eanzenberg »

chimeracoder wrote: post #2

For context, I have a degree in statistics, and I did research with Andrew Gelman (one of the statisticians quoted in the article). Glad to see this is gaining traction! I've been saying this for years: the world would actually be in a better place if we just abandoned p-values altogether. Hypothesis testing is taught in introductory statistics courses because the calculations involved are deceptively easy, whereas m…

If you're doing a measurement, why have a null hypothesis? Alice should sample the population at random, take the height measurements, calculate the average, plot the distribution, calculate the variance. If the distribution is not sufficiently smooth then continue to take measurements until it's smooth, or unchanging. Then Alice is done discovering all there is to know about the height distribution of the population she sampled. Same with Bob.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#13

Post by [deleted] »

carbocation wrote: post #10
eanzenberg wrote: post #5

5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at.

> 5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p This would still end up leading to misuse of P-values. Let's say you're doing a genome-wide association study on several hundred thousand SNPs. The traditional threshold is 5e-8 (0.05 / 1,000,000 effective tests). So using 3e-7 for the threshold for "discovery", you'd count many things as discovery that shouldn't be so. On the other hand, let's sa…

[deleted]

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#14

Post by darawk »

chimeracoder wrote: post #11
darawk wrote: post #8

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

> I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? It is the probability of achieving a result at least as far from the hypothesized value exclusively due to random variation that is uncorrelated with the explanatory variable(s) at hand [and subject to a number of other assumptions].

Ya, but is that not equivalent to the statement 'due to chance'? I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#15

Post by eanzenberg »

carbocation wrote: post #10
eanzenberg wrote: post #5

5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p < 3e-7) for discoveries and reserve 0.05 < p < 3e-7 for stuff we should take closer looks at.

> 5% (1 in 20) is a pretty weak threshold to pass. Let's go 5 sigma (p This would still end up leading to misuse of P-values. Let's say you're doing a genome-wide association study on several hundred thousand SNPs. The traditional threshold is 5e-8 (0.05 / 1,000,000 effective tests). So using 3e-7 for the threshold for "discovery", you'd count many things as discovery that shouldn't be so. On the other hand, let's sa…

Multiple comparisons? 1 million independent tests? Hello?

>Your P value is 0.0002. This doesn't count as discovery, but clinically I know what my judgment is going to be.

>Let's go 5 sigma (p ^Means you should start a new trial with n > 20 given the same placebo/drug split.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#16

Post by tokenadult »

The article submitted here leads to the American Statistical Association statement on the meaning of p values,[1] the first such methodological statement ever formally issued by the association. It's free to read and download. The statement summarizes into these main points, with further explanation in the text of the statement.

"What is a p-value?

"Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (for example, the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.

"Principles

"1. P-values can indicate how incompatible the data are with a specified statistical model.

"2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

"3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.

"4. Proper inference requires full reporting and transparency.

"5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

"6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis."

[1] "The ASA's statement on p-values: context, process, and purpose"

http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016....

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#17

Post by chimeracoder »

darawk wrote: post #14
chimeracoder wrote: post #11

Earlier quoted context omitted.

> I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? It is the probability of achieving a result at least as far from the hypothesized value exclusively due to random variation that is uncorrelated with the explanatory variable(s) at hand [and subject to a number of other assumptions].

Ya, but is that not equivalent to the statement 'due to chance'? I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'.

> I think the common understanding of 'chance' would include 'due to random variation uncorrelated with the explanatory variables'.

Yes, but the problem is that it also includes more than that. One of the many problems with p-values is that people assume that the p-value encodes a lot more information than it actually does.

Another problem: it doesn't really tell you any "new" information. All equality-based null hypotheses are false, and we know this due to continuity theory[0]. Really, all a hypothesis test does is tell us if the sample size is large enough to reflect this knowledge. Literally any null hypothesis can be rejected, as long as the sample size is sufficiently large.

It's worse than that, though. Yes, hypothesis testing doesn't tell us anything about the practical significance of the results. But it also doesn't actually tell us what most people thing it does: a measure of how wrong our hypothesis is. Rejection is a binary state: there's no concept of "strongly rejecting" a null hypothesis[1].

[0] We could reject continuity on the basis that the real world is actually discrete at the quantum level, but a lot of the math underpinning hypothesis testing falls apart if you don't assume continuity, so it's broken either way.

[1] Many people - including statisticians - will sometimes imply this colloquially ("the p-value was .00001, so our hypothesis was completely wrong"). It's not always "wrong", because largely you can measure this concept in other ways, but you cannot do so with a p-value.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#18

Post by jules »

darawk wrote: post #8

Is the p-value really not the probability of your results being due to chance? Is that not a perfectly valid definition of it? I suppose 'chance' is a little hand-wavy, but isn't a p-value just the probability of your data given that your hypothesis is false? Isn't that literally and precisely the probability that they occurred by chance?

That's not what a p-value is. A p-value is the probability of getting by random chance a result at least as extreme as the measurement. This is not the same as the probability that the effect you measured is due to chance. The latter isn't even well defined without additional assumptions.

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#19

Post by danenania »

btilly wrote: post #6

I agree as well! Here is what probability theory teaches us. The proper role of data is to adjust our prior beliefs about probabilities to posterior beliefs through Bayes' theorem. The challenge is how to best communicate this result to people who may have had a wide range of prior beliefs. p-values capture a degree of surprise in the result. Naively, a surprising result should catch our attention and cause us to ret…

Thanks for the very interesting thoughts.

Could you explain a bit how your rule of thumb works and why it's better than p-values? Why is the difference vs. the square root of the max available sample size a meaningful measure?

Re: Statisticians Find They Can Agree: It’s Time to Stop Misusing P-Values

#20

Post by btilly »

chimeracoder wrote: post #2

For context, I have a degree in statistics, and I did research with Andrew Gelman (one of the statisticians quoted in the article). Glad to see this is gaining traction! I've been saying this for years: the world would actually be in a better place if we just abandoned p-values altogether. Hypothesis testing is taught in introductory statistics courses because the calculations involved are deceptively easy, whereas m…

For giggles and grins, my aunt and uncle are an actual example of one of the classic frequentist vs bayesian examples where frequentist statistics says that something utterly irrelevant should matter.

Scenario 1 (true). Bill and Lorena had 7 children. 6 were boys, 1 was a girl. Are they biased towards having one gender? A 2-sided p-value says that there are 16 possibilities of this strength or more, each of which has probability 1/2^7, for a p-value of 2^4/2^7 = 1/2^3 = 0.125. We therefore fail to reject the null hypothesis at a p-value of 0.05.

Scenario 2 (also true): Bill and Lorena decided to have children until they had a boy and a girl. They had 6 boys then a girl. Are they biased towards having one gender? An event this unlikely could only happen if they had 6 of the same gender in a row, which is 2 possibilities of probability 1/2^6, for a probability of 2/2^5 = 1/2^5 = 0.03125. We therefore reject the null hypothesis at a p-value of 0.05.

Now the Bayesian gotcha. According to Bayes' theorem, the intent of Bill and Lorena can have absolutely NO impact on ANY calculation of posterior probabilities from prior expectations. There is no logical way in which this fact should matter at all. And yet it did!

Post reply on HN