Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

P < 0.05 Considered Harmful

simplicityissota.substack.com

Post reply on HN
Print view View original

41–50 of 89 posts

Re: P < 0.05 Considered Harmful

#41

Post by hgomersall »

whimsicalism wrote: post #23
gjm11 wrote: post #15

The article is all about why "0.05" might be a bad value to choose. But, more fundamentally, p is often the wrong thing to be looking at in the first place. 1. Effect sizes. Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one ha…

I feel like a lot of these critiques are just straw-manning p-values consideration. Consider effect sizes - this seems to be a completely different (yes important ) question. Obviously the magnitude of the impact of the drug is important - but it isn't a replacement or "something to look at instead of p-value" because the chance that the results you saw are due to random variation is still important! You can see a ma…

If you're not encoding your prior probability, you're just making stuff up. You almost always have some information, even if it's just sanity bounds.

Re: P < 0.05 Considered Harmful

#43

Post by PeterisP »

the_af wrote: post #13
pphysch wrote: post #10

Earlier quoted context omitted.

Most big games implement "randomness" with "pseudorandomness" in the name of controlling variance of outcome, chopping off the long tail

Is there any game that does NOT use pseudorandom generators? And does this significantly change probabilities?

Some games (both computer games and physical board games) intentionally use "shuffled randomness" where e.g. for percentile fail/success rolls you'd take numbers from 1-100 and use true randomness to shuffle that list; in this way the overall probability is the same, but has a substantially different feel as it's impossible for someone to have bad/good luck throughout the whole game and things like "gambler's fallacy" which are false for actual randomness become true.

Re: P < 0.05 Considered Harmful

#44

Post by whimsicalism »

hgomersall wrote: post #41
whimsicalism wrote: post #23

Earlier quoted context omitted.

I feel like a lot of these critiques are just straw-manning p-values consideration. Consider effect sizes - this seems to be a completely different (yes important ) question. Obviously the magnitude of the impact of the drug is important - but it isn't a replacement or "something to look at instead of p-value" because the chance that the results you saw are due to random variation is still important! You can see a ma…

If you're not encoding your prior probability, you're just making stuff up. You almost always have some information, even if it's just sanity bounds.

Trying to fit your preconception into a mathematically convenient conjugate distribution is not as far afield from making stuff up as people want to believe. Maybe it is better with numerical approaches.

But yes, you usually do have some information and it is net better to encode when possible.

Re: P < 0.05 Considered Harmful

#45

Post by uniqueuid »

spekcular wrote: post #40
uniqueuid wrote: post #39

Earlier quoted context omitted.

I can understand why you see it this way, but still disagree: (1) p-values make significance the target, and thus create incentives for underpowered studies, misspecified analyses, early stopping (monitoring significance while collecting data), and p-hacking. (2) p-values separate crucial pieces of information. It represents a highly specific probability (of the observed data, given the null hypothesis is true), but…

I don't understand. CIs are equivalent to computing a bunch of p-values, by test-interval duality. Should I interpret your points as critiques of simple analyses that only test a single point null of no effect (and go no further)? (I would agree that is bad.)

Yes, I argue that individual p-values (as they are used almost exclusively in numerous disciplines) are bad, and adding more information on effect size and errors are needed. CIs do that by conveying (1) significance (does not include zero), (2) magnitude of effect (mean of CI), and (3) errors/noise (width of CI). That's significantly better than a single p-value (excuse the pun).

Re: P < 0.05 Considered Harmful

#46

Post by begemotz »

Besides the Gigerenzer article mentioned below (there are others by the same author worth reading e.g. 'Mindless Statistics'), I would recommend:

'Moving to a world beyond "p https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1...

as well as the classic article by Jacob Cohen "The Earth is Round (p YMMV, but in certain disciplines still, "statistical analysis" is little more than checking for p-values and applying a binary decision rule.

That is without recognizing the shaky theoretical ground of NHST as practiced.

Re: P < 0.05 Considered Harmful

#47

Post by galkk »

the_af wrote: post #12
galkk wrote: post #5

Earlier quoted context omitted.

Xcom isn’t a good example because the game actively lies to you with displayed probabilities https://youtu.be/l0KEDYFWbVc

Can you elaborate on how XCOM lies? I often suspected this (but you can never be sure, since human intuition is bad at probabilities). Is there hard evidence?

https://youtu.be/l0KEDYFWbVc

Re: P < 0.05 Considered Harmful

#48

Post by bookish »

whimsicalism wrote: post #34
ftxbro wrote: post #32

This post is amazing. When I see a post "P But no, this one isn't from a frequentist, or from a Bayesian. It's from a techbro whose solution isn't any kind of multiple hypothesis correction or getting Bayes-pilled, it's to say "Why not just admit you want something akin to p = 0.25 in the first place?" for 'ship criteria' in the only stats context he appears to know which is A/B testing, talking about Maslow hierarch…

> "Why not just admit you want something akin to p = 0.25 in the first place?" for 'ship criteria' That's culture shock for me - I guess this is why I don't work at startups.

It all depends on what you're experimenting on. I do think there are many teams out there who are (in effect) making decisions with less certainty than this, but they wouldn't want to actually quantify it.

Re: P < 0.05 Considered Harmful

#49

Post by uniqueuid »

bookish wrote: post #36
uniqueuid wrote: post #22

So much has been said about p-values and null hypothesis significance testing (NHST) that one blog post probably won't change anybody's opinion. But I want to recommend the wonderful paper "the null ritual" [1] by Gerd Gigerenzer et al. It shows that a precise understanding of what a p-value means is extremely rare even among statistics lecturers, and, more interestingly, that there have always been fundamentally dif…

Thanks for that. I'll give [1] a read. I'm familiar with [2], and cited one of those papers in the blog. About the stupid or harmful nature of null hypothesis testing in general, what do you recommend instead for decision making and for summarization of uncertainty? In the scenario of large (yet fast moving) organizations where most people will have little stats background.

Thanks, I hope you find Gigerenzer useful. The paper is a bit academic, but he also wrote a couple of nice popular science books on the (mis-)perception of numbers and statistics, those might be useful in a business environment.

For real-world applications outside engineering and academia, I would rely heavily on confidence intervals and/or confidence bands. For example, the packages from easystats [1] in R have quite a few very useful visualization functions, which make it very easy to interpret results of statistical tests. You can even get a textual precise description, but then again, that's intended for papers and not a wider audience.

Apart from that, I would mainly echo recommendations from people like Andrew Gelman, John Tukey, Edward Tufte etc.: Visuals are extremely useful and contain a lot of data. Use e.g. scatterplots with jittered points to show raw data and the goodness of fit. People will intuitively make more of it than of a single p-value.

[1] https://easystats.github.io/easystats/

Re: P < 0.05 Considered Harmful

#50

Post by teddyh »

PeterisP wrote: post #43
the_af wrote: post #13

Earlier quoted context omitted.

Is there any game that does NOT use pseudorandom generators? And does this significantly change probabilities?

Some games (both computer games and physical board games) intentionally use "shuffled randomness" where e.g. for percentile fail/success rolls you'd take numbers from 1-100 and use true randomness to shuffle that list; in this way the overall probability is the same, but has a substantially different feel as it's impossible for someone to have bad/good luck throughout the whole game and things like "gambler's fallacy…

> the overall probability is the same

Only for the very first roll. After that, the outcome becomes more and more predictable for each roll.

Post reply on HN