Most scientists 'can't replicate studies by their peers' • Hacker News Board

Re: Most scientists 'can't replicate studies by their peers'

#211

Post by RodericDay » Thu, Feb 23, 2017, 7:47 PM UTC

virusduck wrote: post #46

tinco wrote: post #37

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated? They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Re…

I think you vastly underestimate how much a lot of scientific studies cost. It would be ideal to be able to have studies replicated by a separate group, but there are many times labs that have specialized equipment and engineering that make replicating studies by some random group unfeasible. In addition, if you spend too much time trying to replicate others' work, you have no time to work on the things that will act…

> I think you vastly underestimate how much a lot of scientific studies cost

Do you have data for this claim? There's tons of extraordinarily expensive experiments through and through, but there's also stuff with incredibly high and time-consuming up-front design and exploration costs that is actually almost trivial to replicate on a per-unit basis.

Re: Most scientists 'can't replicate studies by their peers'

#212

Post by RodericDay » Thu, Feb 23, 2017, 7:49 PM UTC

dekhn wrote: post #148

mdpopescu wrote: post #140

Earlier quoted context omitted.

How can there be any other answer than "yes"? The point of science was that it's reproducible; you are saying that what you did was not; therefore, it wasn't science. What is the difference between your paper and an Uri Geller "experiment"? Both are extremely hard to duplicate for objective reasons so their results have to be accepted based on reputation. (Imagine someone trying to publish the same thing but using "S…

Actually, reproducibility is not considered a necessity in science. It is factually false to say that a person who carries out an experiment in good faith and shares their methods and results is not scientific- they just aren't satisfying a criterion which is a good thing to have. It's pretty clear what's different between my paper and a Uri Geller experiment. If you can't see the difference, you're either being obst…

> Actually, reproducibility is not considered a necessity in science.

By whom? I think you're using as a premise the very matter that is being discussed.

Re: Most scientists 'can't replicate studies by their peers'

#213

Post by daughart » Thu, Feb 23, 2017, 7:55 PM UTC

Direct replication is a waste of time. Move forward, later when there is better technology/data, it will either agree or disagree.

Re: Most scientists 'can't replicate studies by their peers'

#214

Post by [deleted] » Thu, Feb 23, 2017, 7:58 PM UTC

wuschel wrote: post #207

petra wrote: post #193

Earlier quoted context omitted.

Wouldn't it be hard to achieve deep consistency between experiments, in so many labs around the world, with so different conditions/cultures/etc , and when the experimenters aren't experts in consistency , but in their science ? Wouldn't it be better to use something like a cloud biology model - where you define experiments via code, CRO's compete on consistency(and efficiency and automation) and since they probably…

There are already approaches in this direction e.g. such as providing a standardized experimental hardware-software interface with Antha [1]. The complexity of the problems in questions (biological domain, biophysical, biochemical) is daunting - we do not understand many things, " there is plenty of room at the bottom ". [1] https://www.antha-lang.org/

[deleted]

Re: Most scientists 'can't replicate studies by their peers'

#215

Post by snarf21 » Thu, Feb 23, 2017, 8:06 PM UTC

thehardsphere wrote: post #74

An alternative headline would be "Most Published Studies Are Wrong" Am I wrong for considering this not-quite-a-crisis? It has long been the case that initial studies on a topic fail to get reproduced. That's usually because if someone publishes an interesting result, other people do follow up studies with better controls. Sometimes those are to reproduce, sometimes the point is to test something that would follow fr…

I think the hope with these systems is that eventually there will be a preponderance of evidence that proves the study in general terms, not necessarily that everything in this study is 100%. Later scientists will follow along and prove/disprove these findings. If all the studies are 50-50, then we have no idea. If it is 90-10, findings indicate some kernel of truth in these studies. An astute grad student may also be encouraged to look at the 10% that disagreed to see why or if the 90 are too susceptible to selection bias (meaning they decided their conclusion from the beginning and are massaging the data to fit).

The wrong incentives for studies are a bigger problem. I think the only way to solve that is with a higher threshold of peer review to be required before one of these "findings" is put out to the public.

Re: Most scientists 'can't replicate studies by their peers'

#216

Post by Akarnani » Thu, Feb 23, 2017, 8:07 PM UTC

An interesting solve here is structuring research grants for outcome replicability. The article implies a real % of grant money is wasted on results that cannot be replicated.

Could look like X0% of grant money is held for future researchers who work on replicating the outcome. And, wouldn't it be cool if Y% of the grant was held back to be awarded to the researcher whose results were replicated?

This spend, while increasing grant sizes, has the effect of creating replicable science which does more for Science and Society than small grants that end up creating non replicable results.

Re: Most scientists 'can't replicate studies by their peers'

#217

Post by PeterisP » Thu, Feb 23, 2017, 8:09 PM UTC

SilasX wrote: post #180

jonlucc wrote: post #163

Earlier quoted context omitted.

In my job, I work with an outside contract research organization. The way this works is pretty similar to what you're describing, and its a living hell. For starters, it takes 6-10 drafts back and forth to get a protocol to start with, and the resulting document is never shorter than 12 pages. Then, once they start a study, if we get any aberrant data, the question becomes "did they mess that up, or is that data real…

Well, the burden of a policy has to be judged relative to what you're trying to accomplish. If the policy is ensuring that you achieve your ostensible goals, then that burden is justified. Based on your description, it actually sounds like it's making you do things exactly the way science is supposed to work! You quickly identify issues of "real effect or experimenter error or undocumented protocol?" -- and you preve…

The proposal has a problem in that it will increase the quality of results but will enormously slow down progress.

E.g., looking from the perspective of a scientist-reader, if someone has spent a few months doing the ad hoc hands on experiments and achieved interesting observations, then I would want them to publish that research now, instead of spending another half a year to make the repeatable procedure or possibly not publishing it ever because they'd rather do something else than make it up to these standards. There is a benefit from getting all the tricks to make the experiment cleaner, but the clean experiment is just means for acquiring knowledge and ideas for further research, not an end goal by itself for a researcher in that area. Outsiders may have different interests though (https://news.ycombinator.com/item?id=13716233 goes into detail) preferring "finalized and finished research" but to the actual community (who in the end is doing everything and evaluating/judging their peers) generally would prefer the work in progress to be published as-is instead of having more thorough results, but later and less of them.

Outsiders can get the repeatable procedures when it's literally textbook knowledge, packaged in assignments that can be given to students for lab exercises (since that's the level what truly repeatable procedures would require). Insiders want the bleeding edge results now, so they design their journals and conferences to exchange that.

Re: Most scientists 'can't replicate studies by their peers'

#218

Post by notzorbo3 » Thu, Feb 23, 2017, 8:12 PM UTC

bradjohnson wrote: post #184

wycx wrote: post #149

Earlier quoted context omitted.

The way you get funded is by being able to do things that others can't do, or by being better at some technique than everyone else. Giving away all your hard won tricks by putting all of them in a methods section takes away your advantage come funding time.

What's the point of funding something that won't help anyone because the author won't tell you how they did it? How is any conclusion that they've come to useful if nobody can verify if it's correct?

Who says the author(s) won't tell anybody? They only need to keep it secret from the competition, not the hand that feeds them.

Re: Most scientists 'can't replicate studies by their peers'

#219

Post by SilasX » Thu, Feb 23, 2017, 8:16 PM UTC

thearn4 wrote: post #17

zeristor wrote: post #3

Hmmm. Software Engineering has Continuous Integration, since it is so expensive to fix software later in the day. Is there any such thing as Continuous Reproducibility? Constantly checking that the science can be reproduced? How prevalent is this in different branches of Science?

In applied mathematics, the idea of having a standard platform for releasing numerical experiments and standard datasets have come and gone over the years. My advisor said that in the early 2000s, there was a push in some areas to standardize around Java applets for this in a few journals, but never really took hold. Nowadays I would think some form of VM or container technology could probably do the trick while avoi…

>The good news is that you can't really fake proofs or formal analysis.

It's my understanding that most published mathematical proofs aren't "hey look at this theorem in first order logic that we reduced to symbol manipulation"; rather, they present enough evidence that other mathematicians are convinced that such a proof could be constructed.

Is that incorrect?

Re: Most scientists 'can't replicate studies by their peers'

#220

Post by dTal » Thu, Feb 23, 2017, 8:17 PM UTC

wycx wrote: post #111

hannob wrote: post #90

Earlier quoted context omitted.

> I was chatting with a colleague last summer, and asking > him for more details about a method he had published that > I was having trouble reproducing. After pointing me to a > few papers that didn't include the relevant details, > hefinally told me that "if we told everyone about how it > was done, anyone could do it." That's not science, that's bullshit. Can you please expose that? Scientists shouldn't simply get…

That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others. People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...

That might be how something works, but it's too kind to call it "science".