Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It’s not just p=0.048 vs. p=0.052

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

31–40 of 92 posts

Re: It’s not just p=0.048 vs. p=0.052

#31

Post by mruts »

JustFinishedBSG wrote: post #10
mehrdadn wrote: post #5

> To say it again: it is completely consistent with the null hypothesis to see p-values of 0.2 and 0.005 from two replications of the same damn experiment. I don't really follow this. Could someone clarify what is meant here? At what point would this author say something is not consistent with the null hypothesis?

.

I don’t think you understand what a p-value is. The p-value is a percentage output of testing whether a given normal distribution actually has a non-zero mean. It is phrased in terms of the null hypothesis. So a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. Conversely, this means that there is a 95% chance that what you’re testing is true.

Re: It’s not just p=0.048 vs. p=0.052

#32

Post by thanatropism »

tgb wrote: post #21

I think p-values are actually somewhat demonized and I have grown to like them more and more over time. The standard interpretation is actually overly complicated for some reason and it can be simplified to "your p-value cutoff is an upper-bound on the rate of type I errors," over the long term. That's simple and actionable and is an immediate consequence of the definition of p-values! Frankly, I don't know why text-…

P-values are a sub-optimal but okay-ish of quantifying a Popperian hypothesis (a designed-to-be-refutable conjecture). The mathematics is not the problem, the problem is carving science (which in my view (and Quine's and others's) is pretty much defined by the unity of science) in testable morcels.

None of the great achievements of science (Newton, Darwin, Mendeleyev, etc.) were obtained on the basis of Popperian demarcationism/conjectures-and-refutations -- they were obtained by positing a large framework and patching together the empirical case for it.

Re: It’s not just p=0.048 vs. p=0.052

#33

Post by rom1v »

Related, about p-values:

> Here's the problem in a nutshell: If you run 1000 experiments over the course of your career, and you get a significant effect (p > […] However, this is a statement about what happens when the null hypothesis is actually true. In real research, we don't know whether the null hypothesis is actually true. If we knew that, we wouldn't need any statistics! In real research, we have a p value, and we want to know whether we should accept or reject the null hypothesis. The probability of a false positive in that situation is not the same as the probability of a false positive when the null hypothesis is true. It can be way higher.

https://lucklab.ucdavis.edu/blog/2018/4/19/why-i-lost-faith-...

> Here's a more simple thought experiment that gets across the point of why p(null | significant effect) /= p(significant effect | null), and why p-values are flawed as stated in the post.

> Imagine a society where scientists are really, really bad at hypothesis generation. In fact, they're so bad that they only test null hypothesis that are true. So in this hypothetical society, the null hypothesis in any scientific experiment ever done is true. But statistically using a p value of 0.05, we'll still reject the null in 5% of experiments. And those experiments will then end up being published in scientific literature. But then this society's scientific literature now only contains false results - literally all published scientific results are false.

> Of course, in real life, we hope that our scientists have better intuition for what is in fact true - that is, we hope that the "prior" probability in Bayes' theorem, p(null), is not 1.

https://news.ycombinator.com/item?id=16917158

Re: It’s not just p=0.048 vs. p=0.052

#34

Post by dash2 »

JustFinishedBSG wrote: post #7
pacbard wrote: post #2

It looks like the blog author completely missed the point of the statistical significance discussion going on. Most first-tier journals in the social sciences have an acceptance rate of about 5%. At the margins, the differences between acceptance and rejection could be having one more statistical significance result in the table than the paper that was submitted right before or after yours. The problem with a 0.048 a…

I think you should reread the article because it's exactly what the blog author says. Blog author who btw is Andrew Gelman, not just some random guy on Medium, his blog is well well worth reading. Fighting bad stats in science is kind of his hobby/life mission.

Yeah, he had a major part in creating the stats discussion that is going on. I doubt that he has missed its point.

Re: It’s not just p=0.048 vs. p=0.052

#35

Post by [deleted] »

tgb wrote: post #21

I think p-values are actually somewhat demonized and I have grown to like them more and more over time. The standard interpretation is actually overly complicated for some reason and it can be simplified to "your p-value cutoff is an upper-bound on the rate of type I errors," over the long term. That's simple and actionable and is an immediate consequence of the definition of p-values! Frankly, I don't know why text-…

[deleted]

Re: It’s not just p=0.048 vs. p=0.052

#36

Post by firebacon »

mruts wrote: post #31
JustFinishedBSG wrote: post #10

Earlier quoted context omitted.

.

I don’t think you understand what a p-value is. The p-value is a percentage output of testing whether a given normal distribution actually has a non-zero mean. It is phrased in terms of the null hypothesis. So a P < 0.05 means that there is less than a 5% chance that the null hypothesis is true. Conversely, this means that there is a 95% chance that what you’re testing is true.

> So a P See https://en.wikipedia.org/wiki/P-value#Basic%20concepts

Re: It’s not just p=0.048 vs. p=0.052

#37

Post by dash2 »

mehrdadn wrote: post #29
firebacon wrote: post #25

Earlier quoted context omitted.

Check out these links [0] [1] or google for "p value distribution" or "p curve" [0] http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/t... [1] https://en.wikipedia.org/wiki/P-value#Distribution

I'm not following those links either. How is p uniformly distributed under H0? If you assume H0 then obtaining a p-value near 0 is going to be damn impossible. Whereas obtaining one similarly close to 0.5 is going to be ridiculously more likely. Am I severely lacking sleep and going crazy or something? Maybe I should check back in like half a day to see what people have said, I feel like I must be completely confused…

They're right. If the null hypothesis is true, then the probability of getting any p value is equal. Put it another way, the p value is the probability of getting the observed data (or more extreme) under the null. So, under the null, 10% of the time you will get data with a p value of 10% or less; 20% of the time, you will get data with a p value of 20% or less; and so on. And that's the uniform distribution!

Here's an R example to play with:

  pvals
That will plot you a nice uniform line on [0, 1].

(NB: I have no idea why OP talked about p values following a normal distribution. That doesn't make sense to me, and I think the post has been deleted.)

Re: It’s not just p=0.048 vs. p=0.052

#38

Post by lonelappde »

thanatropism wrote: post #32
tgb wrote: post #21

I think p-values are actually somewhat demonized and I have grown to like them more and more over time. The standard interpretation is actually overly complicated for some reason and it can be simplified to "your p-value cutoff is an upper-bound on the rate of type I errors," over the long term. That's simple and actionable and is an immediate consequence of the definition of p-values! Frankly, I don't know why text-…

P-values are a sub-optimal but okay-ish of quantifying a Popperian hypothesis (a designed-to-be-refutable conjecture). The mathematics is not the problem, the problem is carving science (which in my view (and Quine's and others's) is pretty much defined by the unity of science) in testable morcels. None of the great achievements of science (Newton, Darwin, Mendeleyev, etc.) were obtained on the basis of Popperian dem…

Your great achievements in science exclude all real world applications (engineering, pharmaceuticals, etc), where the critical details of a hypothesis don't fit on a t-shirt

Re: It’s not just p=0.048 vs. p=0.052

#39

Post by lmm »

tgb wrote: post #21

I think p-values are actually somewhat demonized and I have grown to like them more and more over time. The standard interpretation is actually overly complicated for some reason and it can be simplified to "your p-value cutoff is an upper-bound on the rate of type I errors," over the long term. That's simple and actionable and is an immediate consequence of the definition of p-values! Frankly, I don't know why text-…

Indeed. People keep saying Bayesianism will solve the replication crisis but as far as I can see it will make it worse: without a hard cut-off you have many more degrees of freedom to "hack" and can push ever more marginal "results".
Post reply on HN