P < 0.05 Considered Harmful
simplicityissota.substack.com
P < 0.05 Considered Harmful
1–10 of 89 posts
Re: P < 0.05 Considered Harmful
#2There is a joke between Mathematicians: The Confidence Interval is nothing more than, the interval between first learning about it, thinking you understood it, and the time it takes until you realize what it really means.
Re: P < 0.05 Considered Harmful
#3The thing that made me realize how ineffective P Also XCOM taught me that 98% != 100%
Re: P < 0.05 Considered Harmful
#42σ is fine, but the benefit of modern technology is that we can tell exactly what standard deviation would be needed for the null hypothesis to randomly generate our results. Particle physics holds itself to an "industry standard" of 5 sigma, for example.
The real conversation to be had is -- what standard deviation will we tolerate? Is this something we'll keep doing, and thus A/B test ourselves into a (perhaps quite horrifying) local minimum based on random noise? Is this a single great experiment? Are lives on the line? Will these results be taken Quite Seriously? Is this a test to say "look, this is worth further investigation"?
I'm no statistician but in my opinion this is the first conversation that needs to be had when doing an experiment: what p / σ levels are satisfactory to claim confidence? p<0.05 is a decent heuristic for some experiments, but not all.
Re: P < 0.05 Considered Harmful
#5The thing that made me realize how ineffective P Also XCOM taught me that 98% != 100%
Xcom isn’t a good example because the game actively lies to you with displayed probabilities
Re: P < 0.05 Considered Harmful
#6Re: P < 0.05 Considered Harmful
#7American Statistical Association also released a statement on pvalues:
https://www.tandfonline.com/doi/epdf/10.1080/00031305.2016.1...
Re: P < 0.05 Considered Harmful
#8Maybe tech industry insiders can tell me this ... but do real people actually make product decisions based solely on p < 0.05? Seems like the author is writing about a contrived problem.
Re: P < 0.05 Considered Harmful
#92σ is fine, but the benefit of modern technology is that we can tell exactly what standard deviation would be needed for the null hypothesis to randomly generate our results. Particle physics holds itself to an "industry standard" of 5 sigma, for example. The real conversation to be had is -- what standard deviation will we tolerate? Is this something we'll keep doing, and thus A/B test ourselves into a (perhaps quit…
It's 5σ for discovery, 2σ for null hypothesis "confirmation".
Re: P < 0.05 Considered Harmful
#10The thing that made me realize how ineffective P Also XCOM taught me that 98% != 100%
Xcom isn’t a good example because the game actively lies to you with displayed probabilities https://youtu.be/l0KEDYFWbVc
Most big games implement "randomness" with "pseudorandomness" in the name of controlling variance of outcome, chopping off the long tail