Replace peer review with “peer replication” (2021)

Maybe add people as special authors/contributors to the original work.

There always seems to be a contingent of people that think that anything less than %100 solution is inadequate so nothing is done. Peer review has proven itself inadequate and people hang on to it tooth and nail. Some disciplines should require replication on everything - I won't name Psychology or Social Sciences in general but the failure to replicate rate for some is unacceptable.

> [...] non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper

Either "peer reviewed" articles describe progress of promising results, or they don't. If they don't the research is effectively ignored (at least until someone finds it promising). So let's consider specifically output that described promising results.

After "peer review" any apparently promising results prompt other groups to build on them by utilizing it as a step or building block.

It can take many failed attempts by independent groups before anyone dares publish the absence of the proclaimed observations, since they may try it over multiple times thinking they must have botched it somewhere.

On paper it sounds more expensive to require independent replication, but only because the costs of replication attempts are hidden until its typically rather late.

Is it really more expensive if the replication attempts are in some sense mandatory?

Or is it perhaps more expensive to pretend science has found a one-shot "peer reviewed" method, resulting in uncoordinated independent reproduction attempts that may go unannounced before, or even after failed replications?

The pseudo-final word, end of line?

What about the "in some sense mandatory" replication? Perhaps roll provable dice for each article, and in-domain sortition to randomly assign replicators. So every scientist would be spending a certain fraction of their time replicating the research of others. The types of acceptable excuses to derelict these duties should be scrutinized and controlled. But some excuses should be very valid, for example conscientious objection. If you are tasked to reproduce some of Dr. Mengele's works, you can cop out on condition that you thoroughly motivate your ethical concerns and objections. This could also bring a lot of healthy criticism to a lot of practices, which is otherwise just ignored an glossed over for fear of future career opportunities.

In the PL field, conferences have started to allow authors to submit packaged artifacts (typically, source code, input data, training data, etc) that are evaluated separately, typically post-review. The artifacts are evaluated by a separate committee, usually graduate students. As usual, everything is volunteer. Even with explicit instructions, it is hard enough to even get the same code to run in a different environment and give the same results. Would "replication" of a software technique require another team to reimplement something from scratch? That seems unworkable.

I can't even imagine how hard it would be to write instructions for another lab to successfully replicate an experiment at the forefront of physics or chemistry, or biology. Not just the specialized equipment, but we're talking about the frontiers of Science with people doing cutting-edge research.

I get the impression that suggestions like these are written by non-scientists who do not have experience with the peer review process of any discipline. Things just don't work like that.

Seems like a great way for "inferior" journals to gain reputation. Counting citations seems a pretty silly formula/hack. How often you say something doesn't affect how true it is.

I remember showing someone raw video of a Safire plasma chamber keeping the ball of plasma lit for several minutes. They said they would need to see a peer reviewed paper. The presumption brought about by the enlightenment era that everyone should get a vote was a mistake.