Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

11–20 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#12
Replication in many fields comes with substantial costs. We are unlikely to see this strategy employed on many/most papers. I agree with other commenters that materials and methodology should be provided in sufficient detail so that others could replicate if desired.

Re: Replace peer review with “peer replication” (2021)

#14
post #2

> the real test of a paper should be the ability to reproduce its findings in the real world. ... > What if all the experiments in the paper are too complicated to replicate? Then you can submit to [the Journal of Irreproducible Results]. Observational science is still a branch of science even if it's difficult or impossible to replicate. Consider the first photographs of a live giant squid in its natural habitat, pu…

I think in some of those cases you have conclusions drawn from raw data that could be replicated or reviewed. For example many teams use the same raw data from Large Colliders, or JWT, or other large science projects to reach competiting conclusions.

Yes in a perfect world we would also replicate the data collection but we do not live in a perfect world

Same is true for Drug Trials, there is always a battle over getting the raw data from drug trails as the companies claim that data is trade secret, so independent verification of drug trails is very expensive but if the FDA required not just the release of redacted conclusions and supporting redacted data but 100% of all data gathered it would be alot better IMO

For example the FDA says it will take decades to release the raw data from the COVID Vaccine trials.. Why... and that is after being forced to do so via a law suit.

Re: Replace peer review with “peer replication” (2021)

#15
I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper.

This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of the preparation that way. But properly replicating a paper will still be a lot of effort, especially when there are any issues and it doesn't work on the first try. Then you have to troubleshoot your experiments and make sure that no mistakes were made. That can add a lot of time to the process.

This is also all work that doesn't benefit the scientists replicating the paper. It only costs them money and time.

If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort. If that works this will indirectly support the original paper even if the following papers don't specifically replicate the original results. Though this part is much more problematic if the following experiments fail, then this will likely remain entirely unpublished. But the solution here unfortunately isn't as simple as just publishing negative results, it take far more work to create a solid negative result than just trying the experiments and abandoning them if they're not promising.

Re: Replace peer review with “peer replication” (2021)

#17

How would you peer-replicate observation of a rare, or unique event, for example in astronomy?

Either get your own telescope and gather your own data, or if only one telescope captured a fleeting event, take that data and see if the analysis turns out the same.

Re: Replace peer review with “peer replication” (2021)

#19
post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

Barriers to publication should be lower for replication studies, I think that’s the main problem.

If someone wants to spend some time replicating something that’s only been described in a paper or two, that is valuable work for the community and should be encouraged. If the person is a PhD student using that as an opportunity to hone their skills, it’s even better. It’s not glamorous, it’s not something entirely new, but it is useful and important. And this work needs to go to normal journals, otherwise there’s just be journals dedicated to replication and their impact factor will be terrible and nobody will care.

Re: Replace peer review with “peer replication” (2021)

#20
I like the idea of splitting "peer review" into two, and then having a citation threshold standard where a field agrees that a paper should be replicated after a certain number of citations. And journals should have a dedicated section for attempted replications.

1. Rebrand peer review as a "readability review" which is what reviewers tend to focus on today.

2. A "replicability statement", a separately published document where reviewers push authors to go into detail about the methodology and strategy used to perform the experiments, including specifics that someone outside of their specialty may not know. Credit NalNezumi ITT

Post reply on HN