Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

P < 0.05 Considered Harmful

simplicityissota.substack.com

Post reply on HN
Print view View original

31–40 of 89 posts

Re: P < 0.05 Considered Harmful

#31

Post by uniqueuid »

spekcular wrote: post #30
gjm11 wrote: post #15

The article is all about why "0.05" might be a bad value to choose. But, more fundamentally, p is often the wrong thing to be looking at in the first place. 1. Effect sizes. Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one ha…

I think if hypothesis testing is understood properly, these objections don't have much teeth. 1. Typically we use p-values to construct confidence intervals, answering the concern about quantifying the effect size. (That is, the confidence interval is the collection of all values not rejected by the hypothesis test.) 2. P-values control type I error. Well-powered designs control type I and type II error. Good control…

Your points are theoretically correct, and probably the reason why many statisticians still regard p-values and HNST favorably.

But looking at the practical application, in particular the replication crisis, specification curve analysis, de facto power of published studies and many more, we see that there is an immense practical problem and p-values are not making it better.

We need to criticize p-values and NHST hard, not because they cannot be used correctly, but because they are not used correctly (and are arguably hard to use right, see the Gigerenzer paper I linked).

Re: P < 0.05 Considered Harmful

#32

Post by ftxbro »

This post is amazing.

When I see a post "P But no, this one isn't from a frequentist, or from a Bayesian. It's from a techbro whose solution isn't any kind of multiple hypothesis correction or getting Bayes-pilled, it's to say "Why not just admit you want something akin to p = 0.25 in the first place?" for 'ship criteria' in the only stats context he appears to know which is A/B testing, talking about Maslow hierarchy and namedropping Hula and Netflix. It's seriously like some Silicon Valley parody. Wait is it actually a satire blog?

Re: P < 0.05 Considered Harmful

#33

Post by reibitto »

kibwen wrote: post #24
galkk wrote: post #5

Earlier quoted context omitted.

Xcom isn’t a good example because the game actively lies to you with displayed probabilities https://youtu.be/l0KEDYFWbVc

I've seen no source that shows that Xcom fudges its displayed hit chances. You may be thinking of Fire Emblem, whose games use a variety of well-documented approaches to fudging their rolls: https://fireemblemwiki.org/wiki/True_hit

I know at least XCOM 2 does on certain difficulties. The aim assist values are directly in the INI files (it fudges the numbers in your favor for lower difficulties). Here are instructions on how to remove the aim assists: https://steamcommunity.com/sharedfiles/filedetails/?id=61799...

Re: P < 0.05 Considered Harmful

#34

Post by whimsicalism »

ftxbro wrote: post #32

This post is amazing. When I see a post "P But no, this one isn't from a frequentist, or from a Bayesian. It's from a techbro whose solution isn't any kind of multiple hypothesis correction or getting Bayes-pilled, it's to say "Why not just admit you want something akin to p = 0.25 in the first place?" for 'ship criteria' in the only stats context he appears to know which is A/B testing, talking about Maslow hierarch…

> "Why not just admit you want something akin to p = 0.25 in the first place?" for 'ship criteria'

That's culture shock for me - I guess this is why I don't work at startups.

Re: P < 0.05 Considered Harmful

#35

Post by spekcular »

uniqueuid wrote: post #31
spekcular wrote: post #30

Earlier quoted context omitted.

I think if hypothesis testing is understood properly, these objections don't have much teeth. 1. Typically we use p-values to construct confidence intervals, answering the concern about quantifying the effect size. (That is, the confidence interval is the collection of all values not rejected by the hypothesis test.) 2. P-values control type I error. Well-powered designs control type I and type II error. Good control…

Your points are theoretically correct, and probably the reason why many statisticians still regard p-values and HNST favorably. But looking at the practical application, in particular the replication crisis, specification curve analysis, de facto power of published studies and many more, we see that there is an immense practical problem and p-values are not making it better. We need to criticize p-values and NHST har…

The items you listed are certainly problems, but p-values don't have much to do with them, as far as I can see. Poor power is an experimental design problem, not a problem with the analysis technique. Not reporting all analyses is a data censoring problem (this is what I understand "specification curve analysis" to mean, based on some Googling - let me know if I misinterpreted). Again, this can't really be fixed at the analysis stage (at least without strong assumptions on the form of the censoring). The replication crisis is a combination of these these two things, and other design issues.

Re: P < 0.05 Considered Harmful

#36

Post by bookish »

uniqueuid wrote: post #22

So much has been said about p-values and null hypothesis significance testing (NHST) that one blog post probably won't change anybody's opinion. But I want to recommend the wonderful paper "the null ritual" [1] by Gerd Gigerenzer et al. It shows that a precise understanding of what a p-value means is extremely rare even among statistics lecturers, and, more interestingly, that there have always been fundamentally dif…

Thanks for that. I'll give [1] a read. I'm familiar with [2], and cited one of those papers in the blog.

About the stupid or harmful nature of null hypothesis testing in general, what do you recommend instead for decision making and for summarization of uncertainty? In the scenario of large (yet fast moving) organizations where most people will have little stats background.

Re: P < 0.05 Considered Harmful

#38

Post by Joel_Mckay »

Still not significant:

https://mchankins.wordpress.com/2013/04/21/still-not-signifi...

Nothing like spending 10 minutes reading a paper to see results which are likely nonsense. However, it pales in comparison to spending 3 weeks trying to replicate popular works... only to find it doesn't generalize... you know that ROC was likely from cooked data-sets confounded with systematic compression artifact errors... likely not harmful, but certainly irritating. lol =)

Re: P < 0.05 Considered Harmful

#39

Post by uniqueuid »

spekcular wrote: post #35
uniqueuid wrote: post #31

Earlier quoted context omitted.

Your points are theoretically correct, and probably the reason why many statisticians still regard p-values and HNST favorably. But looking at the practical application, in particular the replication crisis, specification curve analysis, de facto power of published studies and many more, we see that there is an immense practical problem and p-values are not making it better. We need to criticize p-values and NHST har…

The items you listed are certainly problems, but p-values don't have much to do with them, as far as I can see. Poor power is an experimental design problem, not a problem with the analysis technique. Not reporting all analyses is a data censoring problem (this is what I understand "specification curve analysis" to mean, based on some Googling - let me know if I misinterpreted). Again, this can't really be fixed at t…

I can understand why you see it this way, but still disagree:

(1) p-values make significance the target, and thus create incentives for underpowered studies, misspecified analyses, early stopping (monitoring significance while collecting data), and p-hacking.

(2) p-values separate crucial pieces of information. It represents a highly specific probability (of the observed data, given the null hypothesis is true), but does not include effect size or a comprehensive estimate of uncertainty. Thus, to be useful, p-values need to be combined with effect sizes and ideally simulations, specification curves, or meta-analyses.

Thus my primary problem with p-values is that they are an incomplete solution that is too easy to use incorrectly. Ultimately, they just don't convey enough information in their single summary. CIs, for example, are just as simple to communicate, but much more informative.

Re: P < 0.05 Considered Harmful

#40

Post by spekcular »

uniqueuid wrote: post #39
spekcular wrote: post #35

Earlier quoted context omitted.

The items you listed are certainly problems, but p-values don't have much to do with them, as far as I can see. Poor power is an experimental design problem, not a problem with the analysis technique. Not reporting all analyses is a data censoring problem (this is what I understand "specification curve analysis" to mean, based on some Googling - let me know if I misinterpreted). Again, this can't really be fixed at t…

I can understand why you see it this way, but still disagree: (1) p-values make significance the target, and thus create incentives for underpowered studies, misspecified analyses, early stopping (monitoring significance while collecting data), and p-hacking. (2) p-values separate crucial pieces of information. It represents a highly specific probability (of the observed data, given the null hypothesis is true), but…

I don't understand. CIs are equivalent to computing a bunch of p-values, by test-interval duality. Should I interpret your points as critiques of simple analyses that only test a single point null of no effect (and go no further)? (I would agree that is bad.)
Post reply on HN