Live data from Hacker News

It’s not just p=0.048 vs. p=0.052

statmodeling.stat.columbia.edu

21–30 of 92 posts

Re: It’s not just p=0.048 vs. p=0.052

#21
I think p-values are actually somewhat demonized and I have grown to like them more and more over time. The standard interpretation is actually overly complicated for some reason and it can be simplified to "your p-value cutoff is an upper-bound on the rate of type I errors," over the long term. That's simple and actionable and is an immediate consequence of the definition of p-values! Frankly, I don't know why text-books don't give this as the definition of p-values, and they should reserve "probability of an event at least as extreme conditional upon the null hypothesis" as the thing-you-show-to-prove-it's-a-p-value.

The current trend of saying that "cutting p-values off at a specific value is bad" makes me worry. Now you can argue that your p=0.06 result shouldn't be rejected when really we should probably be pushing for stricter standards rather than inching towards looser ones. It also destroys the nice interpretation of p-values above. P-values were literally made to be cut off - if you want to stop doing that, you need to show me a coherent philosophy of what to do instead.

What I do think is true is the problem you have where part A of the experiment suggests X so you test X more directly in part B with a weaker but more specific test and get p=0.06 and now you can't publish. That's a dumb cutoff, clearly a p=0.06 test is likely to shift our belief towards X so it does nothing but bolster part A. Typically papers do this several times and the marginal 'failure' of one step should not sink the entire ship. This is a case where a Bayesian analysis might be more useful as it can incorporate weak evidence.

But the problem I see often is not that p-values are misused but that they were junk in the first place. For example, the widely-used DESeq2 (as well as some competitors in RNA-seq differential expression analysis) will happily spit out p-values of 10^-100 for an experiment with only four replicates in each of two conditions! There is no way you can get that level of evidence from just four replicates, even if the values are 0,0,0,0 and 1e6,1e6,1e6,1e6. The assumption of normality is reasonable near the mean but gets increasingly inaccurate in the tail, which is exactly where you end up when you do things like sort 30,000 tests by their p-values. In fact taking a p-value cutoff is probably the only reasonable thing to do here - that way you'll ignore the fact that it's absurdly small and just treat it as "small enough".

Re: It’s not just p=0.048 vs. p=0.052

#22

> Also, to get technical for a moment, the p-value is not the “probability of happening by chance.” But we can just chalk that up to a casual writing style. Isn't it though? The probability of this large (or larger) of a variance happening purely by chance[1]? This article is highly critical, but the criticism goes over my head at least. [1] assuming a normally distributed population

I think he was being pedantic about how it's the probability of observations at least as extreme as the ones actually obtained happening if the null hypothesis is true, which is not the same thing as those particular observations happening by chance.

Re: It’s not just p=0.048 vs. p=0.052

#23
post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

Nasim Taleb wrote extensively on this.

> P-values are shown to be extremely skewed and volatile, regardless of the sample size n, and vary greatly across repetitions of exactly same protocols under identical stochastic copies of the phenomenon; such volatility makes the minimum p value diverge significantly from the "true" one. Setting the power is shown to offer little remedy unless sample size is increased markedly or the p-value is lowered by at least one order of magnitude.

https://arxiv.org/abs/1603.07532

Video: https://www.youtube.com/watch?v=8qrfSh07rT0

Re: It’s not just p=0.048 vs. p=0.052

#24

> Also, to get technical for a moment, the p-value is not the “probability of happening by chance.” But we can just chalk that up to a casual writing style. Isn't it though? The probability of this large (or larger) of a variance happening purely by chance[1]? This article is highly critical, but the criticism goes over my head at least. [1] assuming a normally distributed population

I think the part that you have correctly included that people forget or elide is that it's the probability under a specific null hypothesis. So it's a function of what you have chosen for that - normal distributions, a certain parameter value of 0, etc. So this means that a) it's not the probability you'd see in the real world under repeated performance b) it's not the probability under other reasonable null hypotheses. Like maybe under the hypothesis parameter = 0 you get an improbably large p-value, but under parameter = 0.1, or with different assumed underlying distributions, you wouldn't see something so extreme.

I'd guess that the original writer understands this, and that Gelman is only pointing it out because casual readers sometimes don't mentally retain the full baggage that the p-value carries.

Re: It’s not just p=0.048 vs. p=0.052

#25

Earlier quoted context omitted.

.

> It means that if the null hypothesis were true, you expect your p-values to be a random variable contained inside a nice bell shaped normal. 0.005 is dead center so it's very likely but O.2 which seems very unprobable is actually only 1std further, it's well inside the bell curve. I'm completely lost here. How is 0.005 "dead center"? Are you assuming p = 0 is the center? Are there negative p-values I'm not seeing t…

Check out these links [0] [1] or google for "p value distribution" or "p curve"

[0] http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/t...

[1] https://en.wikipedia.org/wiki/P-value#Distribution

Re: It’s not just p=0.048 vs. p=0.052

#26
post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

Nasim Taleb wrote extensively on this. > P-values are shown to be extremely skewed and volatile, regardless of the sample size n, and vary greatly across repetitions of exactly same protocols under identical stochastic copies of the phenomenon; such volatility makes the minimum p value diverge significantly from the "true" one. Setting the power is shown to offer little remedy unless sample size is increased markedly…

Are you sure this answers my question? Note that I was neither asking why p-hacking is bad nor even why p-values are bad.

Re: It’s not just p=0.048 vs. p=0.052

#27
Sure. Apply some fuzzy logic. Highly significant. Somewhat significant. And honestly, we're in an age where somewhat significant can often be bolstered later (in the drug industry) by coupling drugs. It's really time to stop believing everything has to be unifactor. Everything that we care about is multifactor and even slight significance could make a difference if added up.

Re: It’s not just p=0.048 vs. p=0.052

#28

Earlier quoted context omitted.

Nasim Taleb wrote extensively on this. > P-values are shown to be extremely skewed and volatile, regardless of the sample size n, and vary greatly across repetitions of exactly same protocols under identical stochastic copies of the phenomenon; such volatility makes the minimum p value diverge significantly from the "true" one. Setting the power is shown to offer little remedy unless sample size is increased markedly…

Are you sure this answers my question? Note that I was neither asking why p-hacking is bad nor even why p-values are bad.

Yes, it answers your question.

What you call "p-value" is a sample from the "p-value distribution" of your experiment.

Taleb shows you can sample a p-value of 0.05 when the actual "true" p-value is 0.12.

Re: It’s not just p=0.048 vs. p=0.052

#29

Earlier quoted context omitted.

> It means that if the null hypothesis were true, you expect your p-values to be a random variable contained inside a nice bell shaped normal. 0.005 is dead center so it's very likely but O.2 which seems very unprobable is actually only 1std further, it's well inside the bell curve. I'm completely lost here. How is 0.005 "dead center"? Are you assuming p = 0 is the center? Are there negative p-values I'm not seeing t…

Check out these links [0] [1] or google for "p value distribution" or "p curve" [0] http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/t... [1] https://en.wikipedia.org/wiki/P-value#Distribution

I'm not following those links either. How is p uniformly distributed under H0? If you assume H0 then obtaining a p-value near 0 is going to be damn impossible. Whereas obtaining one similarly close to 0.5 is going to be ridiculously more likely.

Am I severely lacking sleep and going crazy or something? Maybe I should check back in like half a day to see what people have said, I feel like I must be completely confused right now because literally nothing I've read so far makes sense to me.

Re: It’s not just p=0.048 vs. p=0.052

#30

Earlier quoted context omitted.

Are you sure this answers my question? Note that I was neither asking why p-hacking is bad nor even why p-values are bad.

Yes, it answers your question. What you call "p-value" is a sample from the "p-value distribution" of your experiment. Taleb shows you can sample a p-value of 0.05 when the actual "true" p-value is 0.12.

I guess I have to sleep on this because at a quick glance I can't really make sense of how it answers my question.

I do find it ironic though that this is so difficult to explain that I apparently have to read a paper to understand it... I would've thought the blog post was trying to explain things in simple terms...

Post reply on HN