Why Most Published Research Findings Are False (2005)
fermatslibrary.com
Why Most Published Research Findings Are False (2005)
1–10 of 42 posts
Re: Why Most Published Research Findings Are False (2005)
#2Please add a [2005] tag.
Re: Why Most Published Research Findings Are False (2005)
#3Please add a [2005] tag.
[deleted]
Re: Why Most Published Research Findings Are False (2005)
#4This is why one has to read papers in context. You can't just read a paper: you have to then go off and read some reviews that cover the surrounding science. Then read some other papers on the topic of interest. Pay attention to dates on papers, as opinions change with time. Note the disagreements.
Reading the scientific literature is reading the moment to moment output of a noisy search algorithm. Any given publication is near meaningless on its own.
This business of reading the context for a specific item of research isn't that hard. If as a layperson you feel up to having an opinion on a specific paper, then you are certainly equipped to do more reading in the field.
Start with review papers, which tend to be a gentler uphill slope, and then fit other papers into what you see there. Take note of the disagreements between reviews, the different emphasis placed on different aspects of the topic. In most scientific fields review papers are usually pretty good at explicitly covering the unknowns and debates of interest. The subtexts and unwritten stuff, such as funding-driven conflicts of research strategy, take longer to figure out. But one has to start somewhere.
https://www.fightaging.org/archives/2009/05/how-to-read-the-...
"The scientific community doesn't produce an output of nice, neat tablets of truth, pronouncements come down from the mountain. It produces theories that are then backed by varying weights of evidence: a theory with a lot of support stands until deposed by new results. But it's not that neat in practice either. The array of theories presently in the making is a vastly complex and shifting edifice of debate, contradictory research results, and opinion. You might compare the output of the scientific community in this sense with the output of a financial market: a staggeringly varied torrent of data that is confusing and overwhelming to the layperson, but which - when considered in aggregate - more clearly shows the way to someone who has learned to read the ticker tape."
Re: Why Most Published Research Findings Are False (2005)
#5The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas.
I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those absolutely stunning results were just intelligently crafted experiments that are not applicable elsewhere.
Basically, my whole thesis is just to show that some paper was wrong and the idea is not applicable in another experiment.
Re: Why Most Published Research Findings Are False (2005)
#6Please add a [2005] tag.
Done. Thanks!
Re: Why Most Published Research Findings Are False (2005)
#7The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…
I often have my PhD students re-implement ideas to gain skills and to verify a method works. It is possible to publish these efforts, but it isn't easy. It typically involves comparing multiple methods on datasets they haven't been tested on before to see how well the results generalize beyond the original paper. It is hard to publish in prestigious venues with this approach, but we have had some success. Replication and comarison makes for a good MS or early PhD project, but later PhD students have to showcase an ability to create new algorithms. In my lab, we try very hard to not cherry pick and to do good work that generalizes.
Re: Why Most Published Research Findings Are False (2005)
#8The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…
Why not arxiv?
Re: Why Most Published Research Findings Are False (2005)
#9The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…
I often have my PhD students re-implement ideas to gain skills and to verify a method works. It is possible to publish these efforts, but it isn't easy. It typically involves comparing multiple methods on datasets they haven't been tested on before to see how well the results generalize beyond the original paper. It is hard to publish in prestigious venues with this approach, but we have had some success. Replication…
Agreed, reimplementation is pretty good for students to learn. Our lab requires everybody to work on an implementation project
Re: Why Most Published Research Findings Are False (2005)
#10The problem I can see right now with scientific community is that, no one pays you to re-implement the same idea just to confirm that it is correct. Every professor wants shiny new innovation from her Ph.D student. No one wants you to experiment the currently published ideas. I'm about to finish my master thesis. I implemented a couple of ideas collected from multiple papers in my thesis, and I can say all of those a…
Yes, that’s true. It’s also because the view is: published == valid.
An idea I had once was to say: the group that repeats or falsifies the results of a paper will be published in the same journal. So this motivate people to piggy back on high profile publications. Unsure if it’ll work though