Please add a [2005] tag.
Done. Thanks!
It's a reprint of an article in PlosMed, where the context is obvious in the original but not here, so the title might need changing too.
Why Most Published Research Findings Are False (2005)
31–40 of 42 posts
Re: Why Most Published Research Findings Are False (2005)
#32It's worth noting that this is talking about research that involves sampling groups of people, or other similar sampling approaches. It is not talking about mathematical results which you can often find in computer science, physics, and elsewhere. Not is it necessarily true all of the physical sciences. It also shouldn't be read as "science is broken and wrong so therefore my opinion should be considered equally." Th…
It's not even talking about biology research, or even most medically related research. While it was a great point to make at the time, far too much has been made of it. Yes, don't p-hack, but it's also better to publish data than to withold it just because there wasn't a positive result. We need better publication mechanisms for data that doesn't have any significant findings.
Err, this 100% applies to biology research, where there is absolutely a problem of small sample sizes, small effects, etc. Our studies (my field is cancer) are routinely vastly underpowered because of the high variability of the datasets we are studying, the low availability of samples, and the large number of variables we collect on these sample.
Also, to a good approximation, everybody p-hacks. Furthermore, the habit of publishing "noteworthy" results (true in every journal, especially true for large impact factor journals) is essentially p-hacking across the entire field. This is a huge problem.
Re: Why Most Published Research Findings Are False (2005)
#33Earlier quoted context omitted.
Not surprised to hear this at all . I can remember being specifically told during both undergrad and grad research seminars 'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'. In social sciences, this is an astoundingly common problem. Then, of course, when I have to defend my comp. exam (for a second time) in front of the faculty, I spe…
>"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'." This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).
If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect? To me, that result doesn't call for adding participants to get to an acceptable p value, it means reevaluate your assumptions, because your basis of understanding is wrong.
@epistasis' using the term 'p-hack' is incredibly apt in this case.
The ability for a study to be deemed "correct" in proving a hypothesis might be true of small samples if you are sufficiently rigorous in designing the experiment, but it could still fall apart at larger sizes - which only proves that part of your hypothesis was not statistically meaningful regardless of the conclusion you reached.
Again, a thorough discussion of limitations around any research should address this.
E.g, "Our findings indicate x, but there is no way to know without further study if y is actually true instead"
Re: Why Most Published Research Findings Are False (2005)
#34This only adds credence to the fact that social survey is not a science. Most of it is glorified
door to door salespeople work. It's not even research.
We have to draw a clear line soon to prevent good-intentioned people from being lumped into shitty science.
Re: Why Most Published Research Findings Are False (2005)
#35The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…
I often have my PhD students re-implement ideas to gain skills and to verify a method works. It is possible to publish these efforts, but it isn't easy. It typically involves comparing multiple methods on datasets they haven't been tested on before to see how well the results generalize beyond the original paper. It is hard to publish in prestigious venues with this approach, but we have had some success. Replication…
> Replication and comparison makes for a good MS or early PhD project
Do you think that might be the solution? That is, to get an MS your final project has to be an attempt at replication, and then a PhD has to be a new contribution. If that became the standard, would it solve a large part of this whole issue?
Re: Why Most Published Research Findings Are False (2005)
#36The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…
To be fair, the professors are like this because the funding agencies---ie, the government and a handful of private groups---essentially refuse to fund duplication research. I thus think that the issue has broader origins than the attitudes of individual faculty.
The review panels that score grants at the National Institutes of Health (NIH) are comprised of professors. (Mostly; being a professor isn't technically required.) If a proposal is scored well by the review panel, it is almost always funded. The National Science Foundation (NSF) works similarly. If (a) the review panels were on board with duplication and (b) people started submitting proposals for duplication research, those proposals would probably get funded.
Edit to clarify: The point is that professors, as a group, already have the power to allocate funds to duplication research if they wished to do so. Blame for a lack of duplication research cannot be pinned solely on the funding agencies.
Re: Why Most Published Research Findings Are False (2005)
#37It's worth noting that this is talking about research that involves sampling groups of people, or other similar sampling approaches. It is not talking about mathematical results which you can often find in computer science, physics, and elsewhere. Not is it necessarily true all of the physical sciences. It also shouldn't be read as "science is broken and wrong so therefore my opinion should be considered equally." Th…
While I agree with the general sentiment that not all science is equal, it’s important to remember that math isn’t science. Science is empirical and involves usage of the _scientific method_ for acquisition of knowledge and math is disqualified. Of course math sits at the foundation of the sciences we are talking about. And truly _proving_ something involves math. That’s not what we are talking about though. And comp…
Beware of bugs in the above code; I have only proved it correct, not tried it. – Donald Knuth
How did people learn about the nature of, say, 1D cellular automata, or the Mandelbrot set? By experimenting, classifying, observing, testing hypotheses etc.
Re: Why Most Published Research Findings Are False (2005)
#38Earlier quoted context omitted.
It's not even talking about biology research, or even most medically related research. While it was a great point to make at the time, far too much has been made of it. Yes, don't p-hack, but it's also better to publish data than to withold it just because there wasn't a positive result. We need better publication mechanisms for data that doesn't have any significant findings.
Err, this 100% applies to biology research, where there is absolutely a problem of small sample sizes, small effects, etc. Our studies (my field is cancer) are routinely vastly underpowered because of the high variability of the datasets we are studying, the low availability of samples, and the large number of variables we collect on these sample. Also, to a good approximation, everybody p-hacks. Furthermore, the hab…
How many of your papers consist of a single statistical test, without validation experiments? Because that's the setup that's described in order for "most published research findings are false."
In reality, I've never seen a biology paper without several lab techniques and orthogonal verifications, with p-values on some but not all of those experiments.
Re: Why Most Published Research Findings Are False (2005)
#39The problem have a name: business.
If research is public, well founded by government, and universities are public entity research is accurate and effective, otherwise it's only a matter of making money quickly and moving on.
Re: Why Most Published Research Findings Are False (2005)
#40Earlier quoted context omitted.
>"'It is highly unlikely that you will find a large number of statistically significant results. The sample sizes are just too small'." This is just a tautology. In social science large sample size = statistical significance (there are only false negatives since the null model being tested is always wrong).
If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect? To me, that result doesn't call for adding participants to get to an acceptable p value, it means reevaluate your assumptions, because your basis of understanding is wrong. @epistasis' using the term 'p-hack' is incredibly apt in this case. The ability for a study to be deemed "correc…
>"If this is true, how does that work if the fundamental conclusion re the relationship of 2 entities is the inverse of what you expect?"
Not quite sure what you are trying to say but this doesnt sound like any statistical model I've seen used in social science.
Usually they assume some distribution (eg normal, or "t") and then assume both samples are taken from that distribution. From that they derive the proportion of times you would see the observed difference in means between samples.