Even if the p < .05 threshold is removed, we’re still left with the important threshold of getting published or not getting published. Judging a paper by the quality of its statistical method won’t be enough to sort the publishable from the unpublishable in top journals. As a result, low p-values will continue to be favored and p-hacking will continue. To me it seems like a consequence of scarcity of academic positions and the tenure system.
Moving to a World Beyond "p < 0.05" (2019)
61–70 of 111 posts
Re: Moving to a World Beyond "p < 0.05" (2019)
#62As someone who has studied genetics on my own for the last twenty years I am very glad to read this editorial. For example, take a population of 100 people, and let us say one of them has gene changes in their Fatty Acid Desaturase genes (FADS1 an d FADS2) that change how important Long Chain Omega 3 Fatty Acids (like from fish) are for them. This happens more often in people from indigenous arctic populations. https…
As a layman who doesn't work with medical studies it always struck me that one of the bits of data that isn't (normally) collected along with everything else is genetic samples of all participants. It should be stored alongside everything else so that if the day comes when genetic testing becomes cheap enough it can be used to provide vastly greater insight into the study's results. Even something as simple as a few…
That throws out anonymity. I don't see this getting approved, or people signing up for such studies, apart from those who don't care that there genetic data gets collected and stored.
Even if there is no name saved with the genetic sample, the bar for identification is low. The genes are even more identifying than a name after all. Worse, it contains deep information about the person.
Re: Moving to a World Beyond "p < 0.05" (2019)
#63Earlier quoted context omitted.
> The 0.05 threshold is indeed arbitrary, but the scientific method is sound. Agreed. A single published paper is not science, a tree data structure of published papers that all build off of each other is science.
Sounds good but is that true? A single unreplicated paper could be science couldn't it? Science is a framework within which there are many things, including theories, mistakes, false negatives, replication failures, etc... Science progresses due to quantity more than quality, it is brute force in some sense that way, but it is more a journey than a destination. You "do" science moreso than you "have" science.
A single brick on the ground, all by itself, is not a wall.
But if you take a lot of bricks and arrange them appropriately, then every single one of those bricks is wall.
In other words, just like the article points out down in the "dos" section, it depends on how you're treating that single unreplicated paper. Are you cherry-picking it, looking at it in isolation, and treating it as if it were definitive all by itself? Or are you considering it within a broader context of prior and related work, and thinking carefully about the strengths, limitations, and possible lacunae of the work it represents?
Re: Moving to a World Beyond "p < 0.05" (2019)
#64Earlier quoted context omitted.
> The 0.05 threshold is indeed arbitrary, but the scientific method is sound. Agreed. A single published paper is not science, a tree data structure of published papers that all build off of each other is science.
Sounds good but is that true? A single unreplicated paper could be science couldn't it? Science is a framework within which there are many things, including theories, mistakes, false negatives, replication failures, etc... Science progresses due to quantity more than quality, it is brute force in some sense that way, but it is more a journey than a destination. You "do" science moreso than you "have" science.
I think you and the OP are agreeing with each other. The issue with a "single unreplicated paper" is exactly the issue you bring up with science as a journey. It's possible that this paper has found a genuine finding or that it is nonsense (people can find isolated published papers supporting almost anything they want even if they don't reflect the scientific consensus), but if no other researchers are even bothering to replicate the findings in it it hasn't joined the journey.
Re: Moving to a World Beyond "p < 0.05" (2019)
#65A p-value of 0.05 just means the observed effect will occur by chance 1 time out of 20.
It means more than that to some people, and it shouldn't.
Re: Moving to a World Beyond "p < 0.05" (2019)
#66The core problem with this is:
- DON'T is very clear and specific. Don't say "Stat-Sig", don't conclude causal effect, don't conclude anything based on p>0.05.
- DO is very vague and unclear. Do be thoughtful, do accept uncertainty, do consider all relevant information.
Obviously, thoughtful consideration of all available information is ideal. But until I get another heuristic for "should I dig into this more?" - I'm just gonna live with my 5-10% FPR, thank you very much.
Re: Moving to a World Beyond "p < 0.05" (2019)
#67I think the main issue with "p < 0.05" can be summed up as people use it to "prove" a phenomenon exists rather than a pragmatic cutoff to screen for interesting things to investigate further.
Yeah but without a hardline how would you decide what to publish?
Publishing only significant results is a terrible idea in the first place. Publishing should be based on how interesting the design of the experiment was, not how interesting the result was.
Re: Moving to a World Beyond "p < 0.05" (2019)
#68My favorite part about pWe're all just fancy monkeys with lightning rocks, it's fine dude
Re: Moving to a World Beyond "p < 0.05" (2019)
#69Seems to me a more reasonable threshold would be at most 10^-9 probability of the conclusion being false (in the Bayesian sense of course), with the prior chosen by the editor and reviewers (and the models being selected upon also being agreeable to them).
Re: Moving to a World Beyond "p < 0.05" (2019)
#70The 0.05 threshold is indeed arbitrary, but the scientific method is sound. A good researcher describes their study, shows their data and lays their own conclusions. There is just no need (nor possibility) of a predefined recipe to resume the study result into a "yes" or a "no". Research is about increasing knowledge; marketing is about labelling.
> The 0.05 threshold is indeed arbitrary, but the scientific method is sound. Agreed. A single published paper is not science, a tree data structure of published papers that all build off of each other is science.
Precisely. As a scientist, that’s how it works.
If a new paper with an outrageous claim pops up, people are automatically suspicious. Until it’s been reproduced by a few labs, it’s just “interesting”.
Then once it’s been validated and new science is built off of it, it’s not really accepted as foundational.