Most scientists 'can't replicate studies by their peers' • Hacker News Board

Re: Most scientists 'can't replicate studies by their peers'

#11

Post by ZeroGravitas » Thu, Feb 23, 2017, 2:28 PM UTC

droopyEyelids wrote: post #5

>The reproducibility difficulties are not about fraud, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge. That would be relatively easy to stamp out. Instead, she says: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about." Well, maybe I'm too much of a layman, but that doesn't qui…

It's more like all the competitors in a market lowering their safety standards to cut costs. If buyers can't accurately assess value then it turn into a bad situation for everyone.

Anyone trying to do the right thing goes out of business and someone cutting corners get's their business.

So a tragedy of the commons style "collective action" problem.

Re: Most scientists 'can't replicate studies by their peers'

#12

Post by tgb » Thu, Feb 23, 2017, 2:29 PM UTC

ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

I don't think that analogy is at all accurate and I think the conclusion that you reach from it is completely incorrect.

Re: Most scientists 'can't replicate studies by their peers'

#13

Post by arca_vorago » Thu, Feb 23, 2017, 2:29 PM UTC

It's because there is a huge problem with the scientific publishing business but because people don't want to admit it because it makes science seem "weak" these issues are largely ignored because of other conflicts of (self)interest.

I'm not a scientist in anything but the colloquial term used as description for a curious and interested person, but when I spent time as a sysadmin at a genetics lab I actually had to read papers as part of the job.

I had previously held "science" up on a pedestal, but I quickly learned that bad science abounds even in reputable publications, and is rarely called out (mostly because scientists use publication to further careers largely based on name-on-paper count).

These days, every time I hear some scientist say "I've been published $largenumber of times," I think to myself 1/3 are probably impossible to reproduce, and 1/3 are probably "I developed this field specialized technique so I get a name drop but didn't actually participate in the study."

Re: Most scientists 'can't replicate studies by their peers'

#14

Post by gcp » Thu, Feb 23, 2017, 2:30 PM UTC

zeristor wrote: post #3

Hmmm. Software Engineering has Continuous Integration, since it is so expensive to fix software later in the day. Is there any such thing as Continuous Reproducibility? Constantly checking that the science can be reproduced? How prevalent is this in different branches of Science?

The problem is that many branches of science don't have any immediate pressure to produce something that is usable by people outside the field, and sometimes not even peers. So they do what is needed to get out papers, and bothering about eliminating false results goes against that interest.

I'm going to bet that in competitive research branches, for practical applications, that have objectively verifiable results, most studies will, in fact, be reproducible.

Re: Most scientists 'can't replicate studies by their peers'

#15

Post by endorphone » Thu, Feb 23, 2017, 2:30 PM UTC

ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

It is a terrible thing, and it is absolutely nothing like finding bugs.

Reproducibility is a core requirement of good science, and if we need to compare it to software engineering, the reproducibility crisis is like the adage "many eyes make all bugs shallow", when the assumption that there is many eyes even looking is often untrue. Most studies are never reproduced, but are held as true under the belief that if someone tried they could.

EDIT: You claimed that in an ideal world, 100% of experiments/studies would not be reproducible. This denotes a profound misunderstanding of the scientific process, or the whole basis of reproducibility. In an idea world, 100% of studies would be vetted through reproduction, and 100% of them would be reproducible. This is essentially the fundamental assumption of the scientific process.

Re: Most scientists 'can't replicate studies by their peers'

#16

Post by eclee2 » Thu, Feb 23, 2017, 2:33 PM UTC

ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

If there are errors in a study's methods that make it unreplicable, then it shouldn't have passed peer review or been published.

Re: Most scientists 'can't replicate studies by their peers'

#17

Post by thearn4 » Thu, Feb 23, 2017, 2:34 PM UTC

zeristor wrote: post #3

Hmmm. Software Engineering has Continuous Integration, since it is so expensive to fix software later in the day. Is there any such thing as Continuous Reproducibility? Constantly checking that the science can be reproduced? How prevalent is this in different branches of Science?

In applied mathematics, the idea of having a standard platform for releasing numerical experiments and standard datasets have come and gone over the years. My advisor said that in the early 2000s, there was a push in some areas to standardize around Java applets for this in a few journals, but never really took hold. Nowadays I would think some form of VM or container technology could probably do the trick while avoiding configuration hell. Commercial licensing for things like MATLAB or COMSOL etc. would be the real challenge for totally open validation in a lot of disciplines. Proprietary software is way more prevalent in scientific and engineering disciplines than I think many general software developers realize.

The good news is that you can't really fake proofs or formal analysis. But the truth is, many folks in the area do cherry pick use case examples/numerical validation as much as you see in other disciplines. Perverse incentives to publish, publish, publish while the tenure clock is ticking keeps this trend going I think.

Re: Most scientists 'can't replicate studies by their peers'

#18

Post by ZeroGravitas » Thu, Feb 23, 2017, 2:34 PM UTC

tgb wrote: post #12

ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

I don't think that analogy is at all accurate and I think the conclusion that you reach from it is completely incorrect.

I think I must have expressed myself poorly, as I think my conclusion is the same as the article suggests i.e. that the science/code shouldn't be considered "done", until it's been peer-reviewed, since it's easy to fool yourself and others if you're not actually reviewing and testing your code.

What did you take away from my comment?

Re: Most scientists 'can't replicate studies by their peers'

#19

Post by pvaldes » Thu, Feb 23, 2017, 2:34 PM UTC

ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

Science publishing is based in a review by peers system. All evident bugs (and many non so evident bugs) should be catched before to appear in a journal. Is totally different to standard journalism.

Re: Most scientists 'can't replicate studies by their peers'

#20

Post by blakesterz » Thu, Feb 23, 2017, 2:36 PM UTC

I have a high level question that I don't see answered, but maybe I missed it?

Are most studies conducted by folks in academia before they get tenure? That is, are most of these results that they're trying to replicate or study done by people who are rather new to doing studies? Is this even possible to know? My guess would be yes, but really I don't have much to base that on. And if that is a yes could that have something to do with the problem here?