Replace peer review with “peer replication” (2021) • Hacker News Board

Re: Replace peer review with “peer replication” (2021)

#31

Post by dongping » Sun, Aug 06, 2023, 3:31 PM UTC

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

While it is a lot of work, I tend to think that one can then always publish preprints if they can't wait for the replication. I don't understand why a published paper should count as an achievement (against tenure or funding) at all before the work is replicated. The current model just creates perverse incentives to encourage lying, P-hacking, and cherry-picking. This would at least work for fields like machine learning.

This is, of course, a naive proposal without too much thought into it. But I was wondering what I would have missed here.

Re: Replace peer review with “peer replication” (2021)

#32

Post by brnaftr361 » Sun, Aug 06, 2023, 3:31 PM UTC

backtoyoujim wrote: post #24

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

Yes it would indeed mean slowing down and having more scientists. It would mean disruption is no longer a useful tool for human development.

It may not be. I would be willing to argue that there was a tipping point and we've long exceeded its boundary - progress and disruption now is just making finding an equilibrium in the future increasingly difficult.

So entering into a paradigm where we test the known space - especially presently - would 1) help reduce cruft; 2) abate undersirable forward progress; 3) train the next generation(s) of scientists to be more diligent and better custodians of the domain.

Re: Replace peer review with “peer replication” (2021)

#33

Post by ebiester » Sun, Aug 06, 2023, 3:32 PM UTC

backtoyoujim wrote: post #24

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

Yes it would indeed mean slowing down and having more scientists. It would mean disruption is no longer a useful tool for human development.

I don't necessarily think it would mean more scientists, but it would mean more expense. You have a moderate number of low impact papers that people are doing for tenure today - papers for the purpose of cranking out papers. We are talking about redirecting efforts but increasing quality of what you have.

Re: Replace peer review with “peer replication” (2021)

#34

Post by petesergeant » Sun, Aug 06, 2023, 3:34 PM UTC

SubiculumCode wrote: post #16

Scientist publishes paper based on ABCD data. Replicator: Do you know how much data I'll need to collect? 11,000 particpants followed across multiple timepoints of MRI scanning. Show me the money.

Definitely something that needs large charitable investment, but charities like that do exist, eg Wellcome Trust

Re: Replace peer review with “peer replication” (2021)

#35

Post by ebiester » Sun, Aug 06, 2023, 3:34 PM UTC

miga wrote: post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

Let's not make perfect be the enemy of good. We may never be able to replicate every field, but we could start many fields today. It means changing our values to make replication as a valid path to tenure and promotion and a required element of Ph.D studies.

Re: Replace peer review with “peer replication” (2021)

#36

Post by abnry » Sun, Aug 06, 2023, 3:36 PM UTC

If scientists are going to complain that's its too hard or too expensive to replicate their studies, then that just shows their work is BS.

Re: Replace peer review with “peer replication” (2021)

#37

Post by Eddy_Viscosity2 » Sun, Aug 06, 2023, 3:36 PM UTC

ebiester wrote: post #30

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewer papers but x number of replications, and you are expected to have x replications in your specialty.) You also create a grant funding section for replication which is then passed on to these independent systems. (You would have to have some sort of randomizat…

It's not easy because it isn't simple. How do get all of the universities to change their incentives to back this?

Re: Replace peer review with “peer replication” (2021)

#38

Post by alsodumb » Sun, Aug 06, 2023, 3:40 PM UTC

abnry wrote: post #36

If scientists are going to complain that's its too hard or too expensive to replicate their studies, then that just shows their work is BS.

Nah, it doesn't. It just shows that it's time consuming and expensive to replicate their studies.

Re: Replace peer review with “peer replication” (2021)

#39

Post by gordian-not » Sun, Aug 06, 2023, 3:41 PM UTC

The incentive should be to clear the way for tenure track

The junior faculty will clear the rotten apples at the top by finding flaws in their research and then will win the tenure that was lost in return

This will create a nice political atmosphere and improve science

Re: Replace peer review with “peer replication” (2021)

#40

Post by kergonath » Sun, Aug 06, 2023, 3:41 PM UTC

eesmith wrote: post #2

> the real test of a paper should be the ability to reproduce its findings in the real world. ... > What if all the experiments in the paper are too complicated to replicate? Then you can submit to [the Journal of Irreproducible Results]. Observational science is still a branch of science even if it's difficult or impossible to replicate. Consider the first photographs of a live giant squid in its natural habitat, pu…

> Who seriously thinks this shouldn't have been published until someone else had been able to replicate the result?

Nobody, obviously. You cannot reproduce a result that hasn’t been published, so no new phenomenon is replicated the moment it is first published. The problem is not the publication of new discoveries, it’s the lack of incentives to confirm them once they’ve been published.

In your example, new observations of giant squids are still massively valuable even if not that novel anymore. So new observations should be encouraged (as I am sure they are).

> Or replicate the results of the flyby of Pluto, or flying a helicopter on Mars?

Well, we should launch another probe anyway. And I am fairly confident we’ll have many instances of aircrafts in Mars’ atmosphere and more data than we’ll know what to do with it. We can also simulate the hell out of it. We’ll point spectrometers and a whole bunch of instruments towards Pluto. These are not really good examples of unreproducible observations.

Besides, in such cases robustness can be improved by different teams performing their own analyses separately, even if the data comes from the same experimental setup. It’s not all black or white. Observations are on a spectrum, some of them being much more reliable than others and replication is one aspect of it.

> How would peer replication be relevant?

How would you know which aspects of the observed phenomena come from particularities of this specific lab? You need more than one instance. You need some kind of statistical and factor analyses. Replication in this instance would not mean setting actual labs on fire on purpose.

It’s exactly like studying car crashes: nobody is going to kill people on purpose, but it is still important to study them so we regularly have new papers on the subject based on events that happened anyway, each one confirming or disproving previous observations.