Most scientists 'can't replicate studies by their peers'
1–10 of 356 posts
Re: Most scientists 'can't replicate studies by their peers'
#2Submitting this because it's come up on HN before and there's a few people who think it's limited to just social psychology or similar. But this report include eg cancer treatments.
Re: Most scientists 'can't replicate studies by their peers'
#3Hmmm.
Software Engineering has Continuous Integration, since it is so expensive to fix software later in the day.
Is there any such thing as Continuous Reproducibility?
Constantly checking that the science can be reproduced?
How prevalent is this in different branches of Science?
Re: Most scientists 'can't replicate studies by their peers'
#4Way for a startup offering alternative / unattached peer reviewing maybe?
Re: Most scientists 'can't replicate studies by their peers'
#5>The reproducibility difficulties are not about fraud, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge.
That would be relatively easy to stamp out. Instead, she says: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about."
Well, maybe I'm too much of a layman, but that doesn't quite seem to add up. Is not calling it fraud about protecting people's egos and saving face?
Or is it like if an accountant completely screwed up all his work and got the numbers wrong, but it was because they were a buffoon- not a fraudster? I guess that would need a different word than fraud.
Re: Most scientists 'can't replicate studies by their peers'
#6"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. "
Ironically, it says this as a bad thing, but in an ideal world this would be 100%.
It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't found any points more in the direction that they're not looking hard enough.
edit: pretty much everyone seems to have taken this the opposite way to how I intended it, but re-reading I can't figure out why that is the case. I'll try to re-phrase:
Science cannot be perfect every time. It's just too complex. This is why you need thorough peer review including reproduction. But if that peer review/reproduction is thorough, then it's going to find problems. When the system is working well, basically everyone will have at some point found a problem in something they are reproducing. This is good because that problem can then be fixed and it will become reproducible or be withdrawn. The current situation is that people don't even look for the problems and no-one can trust results.
edited again to change "peer-reviewed" -> "peer-reviewed including reproduction"
Re: Most scientists 'can't replicate studies by their peers'
#7> The problem, it turned out, was not with Marcus Munafo's science, but with the way the scientific literature had been "tidied up" to present a much clearer, more robust outcome.
I've seen this time and time again while working in neuroscience and hearing the same from friends that are still in those fields.
Data is often thoroughly massaged, outliers left out of reporting and methods tuned to confirm, rather than falsify certain outcomes. It's very demotivating as a PhD student to see very significant results, but when you perform the same study, you don't find reality to be as black and white as published papers.
On this note, the majority of papers is still about reporting significant results, leading to several labs chasing dead ends, as none of them can publish "negative" results.
Re: Most scientists 'can't replicate studies by their peers'
#8imho the headline should be the other way around:
> Most scientific studies cannot be replicated by peers
.. which is more to the point.
Re: Most scientists 'can't replicate studies by their peers'
#9"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…
The whole point of an experiment is to isolate a single variable so you can test a falsifiable statement about it.
It's the exact opposite of building a system, which is what coders do.
Re: Most scientists 'can't replicate studies by their peers'
#10> The problem, it turned out, was not with Marcus Munafo's science, but with the way the scientific literature had been "tidied up" to present a much clearer, more robust outcome. I've seen this time and time again while working in neuroscience and hearing the same from friends that are still in those fields. Data is often thoroughly massaged, outliers left out of reporting and methods tuned to confirm, rather than f…
It gets even worse as if you produce a follow-up paper for an improvement, you're generally expected to produce something better. If the original result doesn't hold up, the only alternative is more even fraud, I mean, data massaging.