Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

21–30 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#21

Post by kergonath »

NalNezumi wrote: post #6
miga wrote: post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

Isn't readability and comprehensibility the job of the editor/journal to check. (after all they're actually paid) maybe not for conference, but peer review is more for checking if the methodology, scope, claim, direction, conclusion and relevances is sound&trustable. At least that's my understanding

The editor is often not the right person to decide based on technical details. Most often, articles they receive anre outside their field of expertise and they don’t really have a way of deciding if a section is comprehensible or not. It’s very difficult for an outsider to know what bit of jargon is redundant and what bit is actually important to make sense of the results. So this bit of readability check falls to the referees.

In theory editors (or rather copyeditors, the editors themselves have to handle too many papers to do this sort of thing) should help with things like style, grammar, and spelling. In practice, quality varies but it is often subpar.

Re: Replace peer review with “peer replication” (2021)

#22

Post by kkylin »

NalNezumi wrote: post #6
miga wrote: post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

Isn't readability and comprehensibility the job of the editor/journal to check. (after all they're actually paid) maybe not for conference, but peer review is more for checking if the methodology, scope, claim, direction, conclusion and relevances is sound&trustable. At least that's my understanding

Highly dependent on journal / field. In mine (mathematics) most associate editors work for free, same as reviwers. The reviewer do all the things you say, and in addition try to ensure readability & novelty. Most journals do have professional copy editing, but that's separate from the content review.

I don't know how refereed conference proceedings work (we don't really use these). The only journals I know of that have professional editors (i.e., editors who are not active researchers themselves) are Nature and affiiliated journals, but someone more knowledgeble should correct me here.

Re: Replace peer review with “peer replication” (2021)

#23

Post by thfuran »

julienreszka wrote: post #10
miga wrote: post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

>Experiments that cost billions to conduct If you can't replicate them it's like they didn't happen anyways

So no experiments have happened because I don't have a lab, and CERN is just an elaborate ruse?

Re: Replace peer review with “peer replication” (2021)

#24

Post by backtoyoujim »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

Yes it would indeed mean slowing down and having more scientists.

It would mean disruption is no longer a useful tool for human development.

Re: Replace peer review with “peer replication” (2021)

#26

Post by kergonath »

julienreszka wrote: post #10
miga wrote: post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

>Experiments that cost billions to conduct If you can't replicate them it's like they didn't happen anyways

It’s a bit more subtle than that. Not all papers are equal and I’d trust an article from a large team where error and uncertainty analysis has been done properly (think the Higgs boson paper) over a handful of dodgy experiments that are barely documented properly.

But yeah, in the grand scheme of things if it hasn’t been replicated, then it hasn’t been proven, but some works are credible on their own.

Re: Replace peer review with “peer replication” (2021)

#27

Post by eesmith »

msla wrote: post #5
eesmith wrote: post #2

> the real test of a paper should be the ability to reproduce its findings in the real world. ... > What if all the experiments in the paper are too complicated to replicate? Then you can submit to [the Journal of Irreproducible Results]. Observational science is still a branch of science even if it's difficult or impossible to replicate. Consider the first photographs of a live giant squid in its natural habitat, pu…

With some of the things, but admittedly not most of the things you mentioned, there's a dataset (somewhere) and some code run on that dataset (somewhere) and replication would mean someone else being able to run that code on that dataset and get the same results. Would this require labs to improve their software environments and learn some new tools? Would this require labs to give up whatever used to be secret sauce…

> someone else being able to run that code on that dataset and get the same results.

I think when people talk about "replicate" they mean something more than that.

The dataset could contain coding errors, and the analysis could contain incorrect formulas and bad modeling. Reproducing a bad analysis, successfully, provide no corrective feedback.

I know for one paper I could replicate the paper's results using the paper's own analysis, but I couldn't replicate the paper's results using my analysis.

> Would this require labs to give up whatever used to be secret sauce? That's. The. Point.

That seems to be a very different Point.

Newton famously published results made from using his secret sauce - calculus - by recasting them using more traditional methods.

In the extreme cas, I could publish the factors for RSA-1024 without publishing my factorization method. "I prayed to God for the answer and He gave them to me." You can verify that result without the secret sauce.

I mean, people use all sorts of methods to predict a protein structure, including manual tweaking guided by intuition and insight gained during a reverie or day-dream (à la Kekulé) which is clearly not reproducible. Yet that final model may be publishable, because it may provide new insight and testable predictions.

Re: Replace peer review with “peer replication” (2021)

#28

Post by mattkrause »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

Longer, even!

Some experiments that study biological development or trained animals can take a year or more of fairly intense effort to start generating data.

Re: Replace peer review with “peer replication” (2021)

#29

Post by kergonath »

NalNezumi wrote: post #8

Imo, A more realistic thing to do is "replicability review" and/or requirement to submit "methodology map" to each paper. The former would be a back and forth between a reviewer that inquire and ask questions (based on the paper) with the goal to reproduce the result , but don't have to actually reproduce it. This is usually good to find out missing details in the paper that the writer just took for granted everyone…

> I've met Bio PHD that have wasted Months of their life tracking up experimental details not mentioned in a paper

Same. Now, when I review manuscripts, I pay much more attention to whether there is enough information to replicate the experiment or simulation. We can put out a paper with wrong interpretations and that’s fine because other people will realise that when doing their own work. We cannot let papers get published if their results cannot be replicated.

> The latter would be the result of the former. Instead of having pages long "appendix" section in the main paper, you produce another document with meticulous details of the experiment/methodology with every stone turned together with an peer reviewer. Stamp it with the peer reviewes name so they can't get away with hand wavy review

Things that take too much space to go in the experimental section should go to a electronic supplementary information document. But then it would be nice if the ESI were appended to the article when we download a PDF because tracking them is a pain in the backside. Some fields are better than others about this, for example in materials characterisation studies it’s very common to have ESI with a whole bunch of data and details.

Large dataset should go to a repository or a dataset journal, that way the method is still peer reviewed and the dataset has a doi and is much easier to re-use. It’s also a nice way of doubling a student’s papers count by the end of their PhD.

> Another atrocious thing I've witnessed is a paper claiming they evaluated their method on a benchmark and if you check the benchmark, the task they evaluated on doesn't exit! They forked the benchmark and made their own task without being clear about it! [2]

That’s just evil!

Re: Replace peer review with “peer replication” (2021)

#30

Post by ebiester »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewer papers but x number of replications, and you are expected to have x replications in your specialty.) You also create a grant funding section for replication which is then passed on to these independent systems. (You would have to have some sort of randomization handled as well.) Replication has to be considered at the same value as original research.

And maybe smaller faculties at R2s pivot to replication hubs. And maybe this is easier for some sections of biology, chemistry and psychology than it is for particle physics. We could start where cost of replication is relatively low and work out the details.

It's completely doable in some cases. (It may never be doable in some areas either.)

Post reply on HN