Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Most scientists 'can't replicate studies by their peers'

bbc.co.uk

Post reply on HN
Print view View original

31–40 of 356 posts

Re: Most scientists 'can't replicate studies by their peers'

#31

Post by therpe1 »

endorphone wrote: post #15
ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

It is a terrible thing, and it is absolutely nothing like finding bugs. Reproducibility is a core requirement of good science, and if we need to compare it to software engineering, the reproducibility crisis is like the adage "many eyes make all bugs shallow", when the assumption that there is many eyes even looking is often untrue. Most studies are never reproduced, but are held as true under the belief that if some…

An experiment should be reproducible. The fact that there exist many experiments that are not reproducible is not a problem and, as the parent post indicates, the fact that these experiments are being discovered as non-reprodicible is a good thing.

This is science at work. The system is error-correcting itself in real-time. We are already seeing laboratories devise new approaches and innovations that will, over time, reduce non-reproducibility errors. The "reproducibility crisis" -- isn't a crisis at all. There's nothing fundamentally wrong with science.

Historians of science will find this era amusing though. When the reproducibility crisis was limited to psychology and the "soft sciences" nobody really took it serious, in fact many people leaped at the opportunity to point out that these sciences aren't "washy." Now, interestingly enough, we're seeing essentially everything but physics (the 'hardest', but really the 'simplest' and most 'stable') science suffer the same fate and this idea of science as a socio-historical process is finally starting to sink in. It only took a good 80 years but here we are.

Re: Most scientists 'can't replicate studies by their peers'

#32

Post by steverb »

YCode wrote: post #26
jobvandervoort wrote: post #7

> The problem, it turned out, was not with Marcus Munafo's science, but with the way the scientific literature had been "tidied up" to present a much clearer, more robust outcome. I've seen this time and time again while working in neuroscience and hearing the same from friends that are still in those fields. Data is often thoroughly massaged, outliers left out of reporting and methods tuned to confirm, rather than f…

For what it's worth I see the same thing in enterprise app development. We've been doing a lot of data visualization and it often happens that someone comes to me with a thinly veiled task that's really to prove this or that person/process is at fault for delaying a project or something. Sometimes though the numbers either don't support their opinion or even show a result they don't like and so inevitably they have m…

Yeah, I used to do a lot of financial reporting for a medical group. It eventually got to the point that after the second "those numbers don't look right" that I started asking what they wanted the numbers to show so I didn't waste any more of my time.

Re: Most scientists 'can't replicate studies by their peers'

#33

Post by ZeroGravitas »

endorphone wrote: post #15
ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

It is a terrible thing, and it is absolutely nothing like finding bugs. Reproducibility is a core requirement of good science, and if we need to compare it to software engineering, the reproducibility crisis is like the adage "many eyes make all bugs shallow", when the assumption that there is many eyes even looking is often untrue. Most studies are never reproduced, but are held as true under the belief that if some…

No, I claimed that all scientist would have had the experience of not reproducing something. Because if they do it a lot, as part of a regular process then they will eventually find something that doesn't work because the original scientist didn't document a step correctly or misread the results or just got lucky due to random chance.

Just like all developers will eventually find a bug in code they code review. This is different from all code they review having bugs.

Re: Most scientists 'can't replicate studies by their peers'

#34

Post by choxi »

I thought the reproducibility crisis was limited to the social sciences and some areas of health/medicine, this is the first article I've seen that claims it is a general problem through all of academia.

The Wiki article on the Reproducibility Crisis cites a Nature survey that makes it seem like the issue is widespread through every industry, including the hard sciences like physics and engineering: https://en.m.wikipedia.org/wiki/Replication_crisis#General

Re: Most scientists 'can't replicate studies by their peers'

#36

Post by marcosdumay »

blakesterz wrote: post #20

I have a high level question that I don't see answered, but maybe I missed it? Are most studies conducted by folks in academia before they get tenure? That is, are most of these results that they're trying to replicate or study done by people who are rather new to doing studies? Is this even possible to know? My guess would be yes , but really I don't have much to base that on. And if that is a yes could that have so…

Most people doing science don't have and probably never will have tenure.

My guess would be that most small studies are set and conducted by people under immediate evaluation. I don't have any good guess about large ones.

Re: Most scientists 'can't replicate studies by their peers'

#37

Post by tinco »

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated?

They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Replicated by XX' badge to each publication to reward any institution that replicated a study.

Funders of studies might improve the quality of the research they paid for by offering replication rewards. I.e. 5% of all funding goes towards institutions who replicated results from research they funded.

Of course there would still be some wrinkles to iron out, but surely we could come up with a nicely balanced solution?

Re: Most scientists 'can't replicate studies by their peers'

#38

Post by benrawk »

I am a social scientist studying human behavior, and this is a huge problem in the field. Myself and my statistician friends who analyze the literature have basically concluded that most extremely "novel" and "surprising" findings in the literature aren't even worth trying to replicate (remember, replications cost money to run, so before you start you have to make some judgment about the likelihood of success.) This is especially true of the "sexiest" sub-topics in the field, like social priming and embodied cognition. If you want to learn more about this, the place to look is Andrew Gelman's blog: http://andrewgelman.com/

Re: Most scientists 'can't replicate studies by their peers'

#40

Post by JumpCrisscross »

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know

Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

Post reply on HN