"Essentially, all models are wrong, but some are useful." --George E.P. Box
P values are not as reliable as many scientists assume (2014)
11–20 of 90 posts
Re: P values are not as reliable as many scientists assume (2014)
#12Great article. I'm not sure that replication itself will solve the problem since Type 1 error rate requires asymptotics. We'd have to run many replications and then show convergence. That'll be broadly cost-prohibitive for all but the most important conclusions. Lower thresholds probably won't do it either. Right now, the only solutions I see are:
a) Baysian methods
b) Fisher's single H hypothesis method
c) Tukey's Exploratory Data Analysis method.
d) All of the above.
Re: P values are not as reliable as many scientists assume (2014)
#13Earlier quoted context omitted.
All of us? Scientific experiments always involve some degree of assumption. I'm most skeptical of those who don't state their assumptions.
[deleted]
How about the assumption that the fundamental constants of the universe are not slowly changing day-to-day?
Re: P values are not as reliable as many scientists assume (2014)
#14Great article. I'm not sure that replication itself will solve the problem since Type 1 error rate requires asymptotics. We'd have to run many replications and then show convergence. That'll be broadly cost-prohibitive for all but the most important conclusions. Lower thresholds probably won't do it either. Right now, the only solutions I see are: a) Baysian methods b) Fisher's single H hypothesis method c) Tukey's E…
Relying on effect measures and their confidence intervals, rather than relying on p-values as a "Yes/No" threshold should likely be on your list, especially as an interim step that should be easy for those who still want p-values to swallow.
Re: P values are not as reliable as many scientists assume (2014)
#15Earlier quoted context omitted.
All of us? Scientific experiments always involve some degree of assumption. I'm most skeptical of those who don't state their assumptions.
[deleted]
First off, there's no need for all-caps. This isn't 4chan.
> The fact that assumptions are considered as some unavoidable, forgivable, intricate part of science is part of what fuels anti-science and politics.
No one here, as far as I can tell, is saying, 'oh well, science is full of assumptions therefore science is invalid.' The problem is not with science in general being valid or invalid, but rather with the sorts of experiments being conducted right now. Studies that are not replicable most of the time are bad science, which is different than science itself being 'bad.'
I'm not saying such faulty experiments and studies are unavoidable and forgivable. Quite the opposite, they're flawed and need to be scrutinized more, not less.
Finally, this is not an incurable problem. Bayesian math is one potential solution. There are others.
Re: P values are not as reliable as many scientists assume (2014)
#16Scientists have to do their work in a system that incentivizes bad science. How many people actually get to do their work in an environment that isn't hostile to them?
a) Are you serious about that second question and b) if so, can we discount thermodynamics in our answer? Otherwise it's kind of boring.
We can restrict ourselves to social factors. Nature isn't hostile, it doesn't have human intentionality like that. Seems to me we make work unpleasant for everyone in the misguided belief that people work harder for it.
Re: P values are not as reliable as many scientists assume (2014)
#17Earlier quoted context omitted.
All of us? Scientific experiments always involve some degree of assumption. I'm most skeptical of those who don't state their assumptions.
[deleted]
I agree that science strives to remove bias from its body of knowledge, but it's absolutely unavoidable for humans to paint their subjective experiences onto it. Humans have to bring their own preconceptions to any scientific experiment, even the medium by which we convey the knowledge is bathed in assumptions about what those words mean. Every facet of human knowledge is premised on how humans experience the universe. Given your definition, I don't think anything we know would be considered a fact.
Re: P values are not as reliable as many scientists assume (2014)
#18Great article. I'm not sure that replication itself will solve the problem since Type 1 error rate requires asymptotics. We'd have to run many replications and then show convergence. That'll be broadly cost-prohibitive for all but the most important conclusions. Lower thresholds probably won't do it either. Right now, the only solutions I see are: a) Baysian methods b) Fisher's single H hypothesis method c) Tukey's E…
I don't see why (e) teaching scientists to be statistically literate so they don't abuse or misunderstand these tests, and/or (f) focusing on reproducible results and shaming researchers with sloppy methodology, wouldn't work. The hypothesis test has known limitations, but it's not clear that we should blame null hypothesis tests for people mis-using them, when researchers untrained in stats are just as likely to mis-use any method you give them.
Re: P values are not as reliable as many scientists assume (2014)
#19From my experience, scientists, -at least in biology, where like in sociology you might have a lot of noise to deal with-, have an internal intuition that a single paper with a significant result does not mean that we have found the truth. The recent study which reported a reproducibility in sociology of about 36% strikes me as pretty accurate. I think the scientific system can work with that. It means that if you bu…
There is no way to know how many people tried to build a follow-up experiment which failed and was not published because the failure to replicate will usually be assumed to be due to some mistake, and even carefully finding p > threshold is not very publishable.
Luckily some disciplines have a journal of negative results :-)
Re: P values are not as reliable as many scientists assume (2014)
#20"Essentially, all models are wrong, but some are useful." --George E.P. Box
The p-value test isn't a model, it's a measure of the significance of an effect in data against random noise.