Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not just p=0.048 vs. p=0.052

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

11–20 of 92 posts

Re: It’s not just p=0.048 vs. p=0.052

#11

Post by mehrdadn »

pacbard wrote: post #8
mehrdadn wrote: post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

He is coming at that conclusion from a Bayesian point of view to statistics. He is seeing the p-value as a random variable that can take values from 0 to 1 and follows some distribution. Under these hypotheses, observing a p-value of 0.20 and 0.005 is completely reasonable even if unlikely. Those are just two draws from a random variable. Edit. Under Bayesian statistics testing the null hypothesis is a moot point as…

I still don't get it. What p-value observations would not be "reasonable" here? It seems to me he's saying that anything between 0 and 1 is completely reasonable, which is a completely pointless statement as I see it.

Re: It’s not just p=0.048 vs. p=0.052

#12

Post by JustFinishedBSG »

pacbard wrote: post #8
mehrdadn wrote: post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

He is coming at that conclusion from a Bayesian point of view to statistics. He is seeing the p-value as a random variable that can take values from 0 to 1 and follows some distribution. Under these hypotheses, observing a p-value of 0.20 and 0.005 is completely reasonable even if unlikely. Those are just two draws from a random variable. Edit. Under Bayesian statistics testing the null hypothesis is a moot point as…

That's not even a bayesian point of view, even in a frequentist setting the p-value is a conditional probability, conditioned on the dataset.

Re: It’s not just p=0.048 vs. p=0.052

#13

Post by paulddraper »

> Also, to get technical for a moment, the p-value is not the “probability of happening by chance.” But we can just chalk that up to a casual writing style.

Isn't it though? The probability of this large (or larger) of a variance happening purely by chance[1]?

This article is highly critical, but the criticism goes over my head at least.

[1] assuming a normally distributed population

Re: It’s not just p=0.048 vs. p=0.052

#14

Post by mehrdadn »

JustFinishedBSG wrote: post #10
mehrdadn wrote: post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

.

> It means that if the null hypothesis were true, you expect your p-values to be a random variable contained inside a nice bell shaped normal. 0.005 is dead center so it's very likely but O.2 which seems very unprobable is actually only 1std further, it's well inside the bell curve.

I'm completely lost here. How is 0.005 "dead center"? Are you assuming p = 0 is the center? Are there negative p-values I'm not seeing that somehow balance the positive ones?

How can a random variable that's strictly between 0 and 1 even follow a bell curve?

Re: It’s not just p=0.048 vs. p=0.052

#15

Post by yk »

pacbard wrote: post #2

It looks like the blog author completely missed the point of the statistical significance discussion going on. Most first-tier journals in the social sciences have an acceptance rate of about 5%. At the margins, the differences between acceptance and rejection could be having one more statistical significance result in the table than the paper that was submitted right before or after yours. The problem with a 0.048 a…

> Most first-tier journals in the social sciences have an acceptance rate of about 5%.

Assuming the null hypothesis, that is precisely our expectation of finding something significant under the p < .05 rule. (That is, assuming that all papers try to falsely reject a true null hypothesis, then we expect 5% of the papers to be successful at that and get published.)

Re: It’s not just p=0.048 vs. p=0.052

#16

Post by krona »

mehrdadn wrote: post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

Well, if there is no effect (the effect-size is zero), two different experimenters will likely see two different p-values regardless of how large an experiment either of them runs.

Re: It’s not just p=0.048 vs. p=0.052

#17

Post by mehrdadn »

krona wrote: post #16
mehrdadn wrote: post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

Well, if there is no effect (the effect-size is zero), two different experimenters will likely see two different p-values regardless of how large an experiment either of them runs.

So by this logic "it is completely consistent with the null hypothesis to see p-values of 0.00000001 and 0.99999999 from two replications of the same damn experiment"?

At which point, what is even the point of this statement?

Re: It’s not just p=0.048 vs. p=0.052

#18

Post by paulddraper »

mehrdadn wrote: post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

I have no idea.

For a identical normal populations, repeating an experiment produces pA coin comes up on the same side 3 times in a row vs 8 times in a row...I have no idea why we should shrug and consider the plausibility of coin bias in these two cases about the same.

---

EDIT: I he means that in the case of an actual difference in the populations, p=0.2 and p=0.005 are both pretty likely outcomes.

When the populations are the same, p=0.2 and p=0.005 are quite different happenings.

This is because p-value methods doesn't worry very much about type II errors.

Re: It’s not just p=0.048 vs. p=0.052

#19

Post by kmm »

JustFinishedBSG wrote: post #10
mehrdadn wrote: post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

.

At least for continuous statistics, the p-value is uniformly distributed when the null hypothesis is true.

Re: It’s not just p=0.048 vs. p=0.052

#20

Post by mehrdadn »

pfortuny wrote: post #9
pacbard wrote: post #8

Earlier quoted context omitted.

He is coming at that conclusion from a Bayesian point of view to statistics. He is seeing the p-value as a random variable that can take values from 0 to 1 and follows some distribution. Under these hypotheses, observing a p-value of 0.20 and 0.005 is completely reasonable even if unlikely. Those are just two draws from a random variable. Edit. Under Bayesian statistics testing the null hypothesis is a moot point as…

What he says is (I gather) worse: those events are only separated be 1.1std deviations, which is little.

I think that's what he's saying too, but what is that supposed to show? Is he arguing against some claim that every interval of 1 standard deviation is equally significant? Did anybody make this claim? So far as I know, nobody considers (say) a 6-sigma effect to be 6 times stronger than a 1-sigma effect...
Post reply on HN