Most scientists 'can't replicate studies by their peers' • Hacker News Board

Re: Most scientists 'can't replicate studies by their peers'

#41

Post by sndean » Thu, Feb 23, 2017, 3:00 PM UTC

JumpCrisscross wrote: post #40

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

This exists (https://www.jove.com/), but, in my experience, even having that doesn't necessarily allow you to replicate the experiment.

Though I should say that I really like JOVE. You can learn a lot.

Re: Most scientists 'can't replicate studies by their peers'

#42

Post by DanBC » Thu, Feb 23, 2017, 3:00 PM UTC

eclee2 wrote: post #16

ZeroGravitas wrote: post #6

"Science is facing a "reproducibility crisis" where more than two-thirds of researchers have tried and failed to reproduce another scientist's experiments, research suggests. " Ironically, it says this as a bad thing, but in an ideal world this would be 100%. It would be like saying "2/3rds of coders have reviewed their colleague's code and found bugs". Since bugs are basically unavoidable, the fact that 1/3 haven't…

If there are errors in a study's methods that make it unreplicable, then it shouldn't have passed peer review or been published.

How do you know it's unreplicable until someone tries to replicate it?

Re: Most scientists 'can't replicate studies by their peers'

#43

Post by graeham » Thu, Feb 23, 2017, 3:01 PM UTC

JumpCrisscross wrote: post #40

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

> The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.

Usually, if you want the full details you look/ ask for the thesis or full report that the journal article is the condensed version of. Most journals also allow supplementary sections for more detailed methods. If its too long, people simply won't read it - already most will just read the abstract and look at figures.

Re: Most scientists 'can't replicate studies by their peers'

#44

Post by JumpCrisscross » Thu, Feb 23, 2017, 3:03 PM UTC

You can back the Center for Open Science, which runs these reproducibility projects here https://cos.io/about/our-sponsors/

Re: Most scientists 'can't replicate studies by their peers'

#45

Post by marcosdumay » Thu, Feb 23, 2017, 3:05 PM UTC

droopyEyelids wrote: post #5

>The reproducibility difficulties are not about fraud, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge. That would be relatively easy to stamp out. Instead, she says: "It's about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about." Well, maybe I'm too much of a layman, but that doesn't qui…

That text is describing this situation:

Context: You are a young scientist, under constant evaluation that may not only fire you, but also invalidate lots of your previous work. You have to produce flashy results to grow up on your career. You _really_ do not desire to get stuck where you are now.

Now, let's decide on a new experiment to proceed. Do you choose:

a - Boring important experiment that you can't hype but will surely advance your field;

b - Flashy, risky experiment that probably won't lead anywhere, but will change your life if you got lucky;

Now, let's say you go with "b" (that's a non-brainier). Four years into the experiment your results aren't going anywhere. Do you:

a - Accept it's a failed experiment, accept the failure that will set your career back 4 years, start again;

b - Insist on getting more data. Insist on getting more data. Insist on ... oh, never mind, that last data is impressive¹, publish it and go ahead.

1 - https://www.xkcd.com/882/

Re: Most scientists 'can't replicate studies by their peers'

#46

Post by virusduck » Thu, Feb 23, 2017, 3:06 PM UTC

tinco wrote: post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

I think you vastly underestimate how much a lot of scientific studies cost. It would be ideal to be able to have studies replicated by a separate group, but there are many times labs that have specialized equipment and engineering that make replicating studies by some random group unfeasible.

In addition, if you spend too much time trying to replicate others' work, you have no time to work on the things that will actually continue to fund your own research.

The best thing is to have a healthy, skeptical, but collegial, competition in the field. That still requires more funding though!

Re: Most scientists 'can't replicate studies by their peers'

#47

Post by fsloth » Thu, Feb 23, 2017, 3:07 PM UTC

nacc wrote: post #28

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Secondly, I wish people say: "I replicated the methods and got a solid negative result" instead of "I can't replicate this experiment". Because most of the t…

So why aren't mouse studies done in acoustically isolated rooms? Too expensive or lack of forethought? I know metrological laboratories that are mechanically insulated with all sorts of contraptions making the space vibration free so it'd not as if buildings could not be adapted for their intended scientific use as a matter of principle.

Re: Most scientists 'can't replicate studies by their peers'

#48

Post by graeham » Thu, Feb 23, 2017, 3:07 PM UTC

tinco wrote: post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

This would be a pretty big step - right now there is a big incentive to be first to publish. Often the researchers who would be doing the replication are competitors, who are racing to publish their results first.

Part of publishing is to put it out for discussion. It should be the best effort and thought to be true, but science is about the evolution of knowledge. Very few things are completely understood, especially in biological and micro/nano fields.

Re: Most scientists 'can't replicate studies by their peers'

#49

Post by neffy » Thu, Feb 23, 2017, 3:10 PM UTC

choxi wrote: post #34

I thought the reproducibility crisis was limited to the social sciences and some areas of health/medicine, this is the first article I've seen that claims it is a general problem through all of academia. The Wiki article on the Reproducibility Crisis cites a Nature survey that makes it seem like the issue is widespread through every industry, including the hard sciences like physics and engineering: https://en.m.wiki…

Oh it's general. Apart from anything else, a majority of all scientific papers rely on some form of software, be it simple R statistical scripts, to complex programs. None of those programs ever have bugs in them, right?

Then to be fair to the authors, scientific papers follow a fairly fixed format, often with hard limits on paper length (cf. Nature, etc.). Putting all the detail necessary for replication, and a decent literature review, and an overview , is simple not possible.

And then the meta-problem - which is general - every aspect of science from hiring to grant writing, to managing a phd. farm, to goofing off on ycombinator..., essentially works against anybody trying to do detailed, methodical, provable work, no matter how brilliant they are, because doing all aspects of science properly is incredibly time consuming in a world with an ever reducing attention span.

Re: Most scientists 'can't replicate studies by their peers'

#50

Post by dekhn » Thu, Feb 23, 2017, 3:11 PM UTC

So. Having been a scientist, I observed an interesting phenomenon, several times. It's almost as if scientists enjoy leaving out critical details from the methods section, and other scientists enjoy puzzling out what the missing details are. I think there's this sort-of assumption of competence that for any reasonably interesting paper, the people in the field who are reading it, they have the level of skill to reproduce it even with missing information.