Replace peer review with “peer replication” (2021)
blog.everydayscientist.com
Replace peer review with “peer replication” (2021)
1–10 of 351 posts
Re: Replace peer review with “peer replication” (2021)
#2> the real test of a paper should be the ability to reproduce its findings in the real world. ...
> What if all the experiments in the paper are too complicated to replicate? Then you can submit to [the Journal of Irreproducible Results].
Observational science is still a branch of science even if it's difficult or impossible to replicate.
Consider the first photographs of a live giant squid in its natural habitat, published in 2005 at https://royalsocietypublishing.org/doi/10.1098/rspb.2005.315... .
Who seriously thinks this shouldn't have been published until someone else had been able to replicate the result?
Who thinks the results of a drug trial can't be published until they are replicated?
How does one replicate "A stellar occultation by (486958) 2014 MU69: results from the 2017 July 17 portable telescope campaign" at https://ui.adsabs.harvard.edu/abs/2017DPS....4950403Z/abstra... which required the precise alignment of a star, the trans-Neptunian object 486958 Arrokoth, and a region in Argentina?
Or replicate the results of the flyby of Pluto, or flying a helicopter on Mars?
Here's a paper I learned about from "In The Pipeline"; "Insights from a laboratory fire" at https://www.nature.com/articles/s41557-023-01254-6 .
"""Fires are relatively common yet underreported occurrences in chemical laboratories, but their consequences can be devastating. Here we describe our first-hand experience of a savage laboratory fire, highlighting the detrimental effects that it had on the research group and the lessons learned."""
How would peer replication be relevant?
Re: Replace peer review with “peer replication” (2021)
#3Re: Replace peer review with “peer replication” (2021)
#4Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper.
Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers.
What could be done is to add metadata about papers that were replicated.
Re: Replace peer review with “peer replication” (2021)
#5> the real test of a paper should be the ability to reproduce its findings in the real world. ... > What if all the experiments in the paper are too complicated to replicate? Then you can submit to [the Journal of Irreproducible Results]. Observational science is still a branch of science even if it's difficult or impossible to replicate. Consider the first photographs of a live giant squid in its natural habitat, pu…
With some of the things, but admittedly not most of the things you mentioned, there's a dataset (somewhere) and some code run on that dataset (somewhere) and replication would mean someone else being able to run that code on that dataset and get the same results.
Would this require labs to improve their software environments and learn some new tools? Would this require labs to give up whatever used to be secret sauce? That's. The. Point.
Re: Replace peer review with “peer replication” (2021)
#6Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.
Isn't readability and comprehensibility the job of the editor/journal to check. (after all they're actually paid) maybe not for conference, but peer review is more for checking if the methodology, scope, claim, direction, conclusion and relevances is sound&trustable.
At least that's my understanding
Re: Replace peer review with “peer replication” (2021)
#7Peer Review is the right solution to the wrong problem: https://open.substack.com/pub/experimentalhistory/p/science-...
On replication, it is a worthwhile goal but the career incentives need to be there. I think replicating studies should be a part of the curriculum in most programs - a step toward getting a PhD in lieu of one of the papers.
Re: Replace peer review with “peer replication” (2021)
#8Imo, A more realistic thing to do is "replicability review" and/or requirement to submit "methodology map" to each paper.
The former would be a back and forth between a reviewer that inquire and ask questions (based on the paper) with the goal to reproduce the result, but don't have to actually reproduce it. This is usually good to find out missing details in the paper that the writer just took for granted everyone in the field knows (I've met Bio PHD that have wasted Months of their life tracking up experimental details not mentioned in a paper)
The latter would be the result of the former. Instead of having pages long "appendix" section in the main paper, you produce another document with meticulous details of the experiment/methodology with every stone turned together with an peer reviewer. Stamp it with the peer reviewes name so they can't get away with hand wavy review.
I've read too many papers where important information to reproduce the result is omitted. (for ML/RL) If the code is included I've countless of times found implementation details that is not mentioned in the paper. In matter of fact, there's even results suggesting that those details are the make or break of certain algorithms. [1] I've also seen breaking details only mentioned in code comments...
Another atrocious thing I've witnessed is a paper claiming they evaluated their method on a benchmark and if you check the benchmark, the task they evaluated on doesn't exit! They forked the benchmark and made their own task without being clear about it! [2]
Shit like this make me lose faith in certain science directions. And I've seen a couple of junior researcher giving it all up because they concluded it's all just house of cards.
[1] https://arxiv.org/abs/2005.12729
[2] https://arxiv.org/abs/2202.02465
Edit: also if you think that's too tedious/costly, reminder that publishers rake in record profits so the resources are already there https://youtu.be/ukAkG6c_N4M
Re: Replace peer review with “peer replication” (2021)
#9> the real test of a paper should be the ability to reproduce its findings in the real world. ... > What if all the experiments in the paper are too complicated to replicate? Then you can submit to [the Journal of Irreproducible Results]. Observational science is still a branch of science even if it's difficult or impossible to replicate. Consider the first photographs of a live giant squid in its natural habitat, pu…
With some of the things, but admittedly not most of the things you mentioned, there's a dataset (somewhere) and some code run on that dataset (somewhere) and replication would mean someone else being able to run that code on that dataset and get the same results. Would this require labs to improve their software environments and learn some new tools? Would this require labs to give up whatever used to be secret sauce…
In practice this is happening in many disciplines, for most research, on a daily basis. What _isn't_ happening is that the results of these replications are being independently peer reviewed, because that isn't incentivized. However, when replication fails for whatever reason, it usually leads to insights that themselves lead to stronger scientific work and better publications later on.
Re: Replace peer review with “peer replication” (2021)
#10Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.
>Experiments that cost billions to conduct
If you can't replicate them it's like they didn't happen anyways