Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

71–80 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#71

Post by i_no_can_eat »

dongping wrote: post #31
fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

While it is a lot of work, I tend to think that one can then always publish preprints if they can't wait for the replication. I don't understand why a published paper should count as an achievement (against tenure or funding) at all before the work is replicated. The current model just creates perverse incentives to encourage lying, P-hacking, and cherry-picking. This would at least work for fields like machine learn…

and in this proposal, who will be tasked with replicating the work?

Re: Replace peer review with “peer replication” (2021)

#72

Post by eesmith »

msla wrote: post #59
eesmith wrote: post #27

Earlier quoted context omitted.

> someone else being able to run that code on that dataset and get the same results. I think when people talk about "replicate" they mean something more than that. The dataset could contain coding errors, and the analysis could contain incorrect formulas and bad modeling. Reproducing a bad analysis, successfully, provide no corrective feedback. I know for one paper I could replicate the paper's results using the pape…

My point is that we can, apparently, improve the baseline expectations in the parts of science where this kind of reproducibility is possible. That isn't all science, granted, but it is some science. It isn't a panacea, granted, but it could guard against some forms of misconduct or honest error some of the time. The self-correcting part of science only works when there's something for it to work on, so open data and…

Understood.

But my point is this linked-to essay appears not only to exclude some areas of good science, but to suggest that any topics which cannot be replicated before publication is only worthy of publication in the Journal of Irreproducible Results.

I gave examples to highlight why I disagree with author's opinion.

Please do not interpret this to mean I do not think improvement is possible.

Re: Replace peer review with “peer replication” (2021)

#73

Post by Levitz »

abnry wrote: post #42
alsodumb wrote: post #38

Earlier quoted context omitted.

Nah, it doesn't. It just shows that it's time consuming and expensive to replicate their studies.

If that's the case, then don't claim confidence in the work or make policy decisions based off of it. If there is no epistemological humility, then yes, it is still BS.

If any study costs X, the study and the replication costs somewhere in the ballpark of 2*X. This is not trivial.

Re: Replace peer review with “peer replication” (2021)

#74

Post by kergonath »

eesmith wrote: post #68
kergonath wrote: post #40

Earlier quoted context omitted.

> Who seriously thinks this shouldn't have been published until someone else had been able to replicate the result? Nobody, obviously. You cannot reproduce a result that hasn’t been published, so no new phenomenon is replicated the moment it is first published. The problem is not the publication of new discoveries, it’s the lack of incentives to confirm them once they’ve been published. In your example, new observati…

> Nobody, obviously. You cannot reproduce a result that hasn’t been published, .. The problem is not the publication of new discoveries, it’s the lack of incentives to confirm them once they’ve been published. Your comment concerns post-publication peer-replication, yes? If so, it's a different topic. The linked-to essay specifically proposes: ""Instead of sending out a manuscript to anonymous referees to read and re…

> That's pre-publication peer-replication, and my comment was only meant to be interpreted in that light.

Sorry I might have gone mixed up between threads.

Yeah, pre-publication replication is nice (I do it when I can and am suspicious of some simulation results), but is not practical at scale. Besides, the role of peer review is not to ensure results are right, that is just not sustainable for referees.

Re: Replace peer review with “peer replication” (2021)

#75

Post by kergonath »

s1artibartfast wrote: post #44
kergonath wrote: post #19

Earlier quoted context omitted.

Barriers to publication should be lower for replication studies, I think that’s the main problem. If someone wants to spend some time replicating something that’s only been described in a paper or two, that is valuable work for the community and should be encouraged. If the person is a PhD student using that as an opportunity to hone their skills, it’s even better. It’s not glamorous, it’s not something entirely new,…

They're basically no barriers to publication. There are a number of normal journals that publish everything submitted if it appears to be honest research.

Not nice journals, though. At least not in my experience but that’s probably very field-dependent. It’s not uncommon to get a summary rejection letter for lack of novelty and that is one aspect they stress when they ask us to review articles.

Re: Replace peer review with “peer replication” (2021)

#76

Post by waynecochran »

I spent a lot of my graduate years in CS implementing the details of papers only to learn that, time and time again, the paper failed to mention all the short comings and fail cases of the techniques. There are great exceptions to this.

Due to the pressure of "publish or die" there is very little honesty in research. Fortunately there are some who are transparent with their work. But for the most part, science is drowning in a sea of research that lacks transparency and replication short falls.

Re: Replace peer review with “peer replication” (2021)

#77

Post by azan_ »

abnry wrote: post #36

If scientists are going to complain that's its too hard or too expensive to replicate their studies, then that just shows their work is BS.

What if it REALLY is too expensive? You do realize that there are studies which literally cost millions of dollars? Getting funding for original studies is hard enough, good luck securing additional funds for replication.

Re: Replace peer review with “peer replication” (2021)

#78

Post by analog31 »

infogulch wrote: post #20

I like the idea of splitting "peer review" into two, and then having a citation threshold standard where a field agrees that a paper should be replicated after a certain number of citations. And journals should have a dedicated section for attempted replications. 1. Rebrand peer review as a "readability review" which is what reviewers tend to focus on today. 2. A "replicability statement", a separately published docu…

Every experimental paper I've ever read has contained an "Experimental" section, where they provide the details on how they did it. Those sections tend to be general enough, albeit concise.

In some fields, aside from specialized knowledge, good experimental work requires what we call "hands." For instance, handling air sensitive compounds, or anything in a condensed or crystalline state. In my thesis experiment, some of the equipment was hand made, by me.

Sometimes specialized facilities are needed. My doctoral thesis project used roughly 1/2 million dollars of gear, and some of the equipment that I used was obsolete and unavailable by the time I finished.

Re: Replace peer review with “peer replication” (2021)

#79

Post by snitty »

abnry wrote: post #36

If scientists are going to complain that's its too hard or too expensive to replicate their studies, then that just shows their work is BS.

>If scientists are going to complain that's its too hard or too expensive to replicate their studies, then that just shows their work is BS.

1 mg of anti-rabbit antibody (a common thing to use in a lot of biology experiments) is $225 [1]. Outside of things like standard buffers and growth medium for prokaryotes, this is going to be the cheapest thing you use in an experiment.

1/10th of that amount for anti-flagellin antibody is $372. [2]

A kit to prep a cell for RNA sequencing is $6-10 per use. That's JUST isolation of the RNA. Not including reverse transcribing it to cDNA for sequencing, or the sequencing itself. [3]

Let's not even reach things like materials science where you may be working on an epitaxial growth paper, and there are only a handful of labs where they could even feasibly repeat the experiment.

Or say something with a BSL-3 lab where there are literally only 15 labs in the US that could feasibly do the work, assuming they aren't working on their own stuff. [4]

[1] - https://www.thermofisher.com/antibody/product/Goat-anti-Rabb... [2] https://www.invivogen.com/anti-flagellin [3] https://www.thermofisher.com/order/catalog/product/12183018A [4] https://www.niaid.nih.gov/research/tufts-regional-biocontain...

Re: Replace peer review with “peer replication” (2021)

#80

Post by SubiculumCode »

petesergeant wrote: post #34
SubiculumCode wrote: post #16

Scientist publishes paper based on ABCD data. Replicator: Do you know how much data I'll need to collect? 11,000 particpants followed across multiple timepoints of MRI scanning. Show me the money.

Definitely something that needs large charitable investment, but charities like that do exist, eg Wellcome Trust

Like 290+ million, just to get started.
Post reply on HN