Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

61–70 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#61

Post by brightball »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper.

The alternative is a bunch of stuff being published which people belief as "science" that doesn't hold up under scrutiny, which undermines the reliability of science itself. The current approach simply gives people reason to be skeptical.

Re: Replace peer review with “peer replication” (2021)

#62

Post by MichaelZuo »

ebiester wrote: post #55
Eddy_Viscosity2 wrote: post #37

Earlier quoted context omitted.

It's not easy because it isn't simple. How do get all of the universities to change their incentives to back this?

We agree - the "simple not easy" turn of phrase is speaking to that point. It is easy once implemented, but it isn't easy to transition. (I am academia-adjacent by marriage but closer to the humanities, so I understand the amount of work it would take to perform the transition.)

This isn't just not easy, it would probably be extremely political to change the structure of the NSF, National Labs, all universities and colleges, etc., so dramatically.

Re: Replace peer review with “peer replication” (2021)

#63

Post by nine_k »

elashri wrote: post #45

Great, but who is going to fund the peer replication?. The economics of research now doesn't even provide a compensation for peer review process time.

Maybe the numerous complaints about the crisis of science are somehow related to the fact that scientific work is severely underpaid.

The pay difference between research and industry in many areas is not even funny.

Re: Replace peer review with “peer replication” (2021)

#64

Post by mathisfun123 »

miga wrote: post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

>Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper.

I have had a paper rejected twice in a row over the last year. Both times the comments include something like "paper was very well-wriiten; well-written enough that an undergrad could read it".

Peer review ensures the gates are kept.

Re: Replace peer review with “peer replication” (2021)

#65

Post by fodkodrasz »

abnry wrote: post #36

If scientists are going to complain that's its too hard or too expensive to replicate their studies, then that just shows their work is BS.

I guess if software developers will complain that it's too hard or too expensive to thoroughly test their code to ensure exactly zero bugs at release[1], then that just shows their work is BS.

[1]: if you have delivered telco code to Softbank you may have heard this sentence

Re: Replace peer review with “peer replication” (2021)

#67

Post by jofer »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

Also, don't forget that a lot of replication would fundamentally involve going and collecting additional samples / observations / etc in the field area, which is often expensive, time consuming, and logistically difficult.

It's not just "can we replicate the analysis on sample X", but also "can we collect a sample similar to X and do we observe similar things in the vicinity" in many cases. That alone may require multiple seasons of rather expensive fieldwork.

Then you have tens to hundreds of thousands of dollars in instrument time to pay to run various analysis which are needed in parallel with the field observations.

It's rarely the simple data analysis that's flawed and far more frequently subtle issues with everything else.

In most cases, rather than try to replicate, it's best to test something slightly different to build confidence in a given hypothesis about what's going on overall. That merits a separate paper and also serves a similar purpose.

E.g. don't test "can we observe the same thing at the same place?", and instead test "can we observe something similar/analogous at a different place / under different conditions?". That's the basis of a lot of replication work in geosciences. It's not considered replication, as it's a completely independent body of work, but it serves a similar purpose (and unlike replication studies, it's actually publishable).

Re: Replace peer review with “peer replication” (2021)

#68

Post by eesmith »

kergonath wrote: post #40
eesmith wrote: post #2

> the real test of a paper should be the ability to reproduce its findings in the real world. ... > What if all the experiments in the paper are too complicated to replicate? Then you can submit to [the Journal of Irreproducible Results]. Observational science is still a branch of science even if it's difficult or impossible to replicate. Consider the first photographs of a live giant squid in its natural habitat, pu…

> Who seriously thinks this shouldn't have been published until someone else had been able to replicate the result? Nobody, obviously. You cannot reproduce a result that hasn’t been published, so no new phenomenon is replicated the moment it is first published. The problem is not the publication of new discoveries, it’s the lack of incentives to confirm them once they’ve been published. In your example, new observati…

> Nobody, obviously. You cannot reproduce a result that hasn’t been published, .. The problem is not the publication of new discoveries, it’s the lack of incentives to confirm them once they’ve been published.

Your comment concerns post-publication peer-replication, yes?

If so, it's a different topic. The linked-to essay specifically proposes:

""Instead of sending out a manuscript to anonymous referees to read and review, preprints should be sent to other labs to actually replicate the findings. Once the key findings are replicated, the manuscript would be accepted and published.""

That's pre-publication peer-replication, and my comment was only meant to be interpreted in that light.

Re: Replace peer review with “peer replication” (2021)

#69

Post by vinnyvichy »

leedrake5 wrote: post #7

Peer Review is the right solution to the wrong problem: https://open.substack.com/pub/experimentalhistory/p/science-... On replication, it is a worthwhile goal but the career incentives need to be there. I think replicating studies should be a part of the curriculum in most programs - a step toward getting a PhD in lieu of one of the papers.

Fear of the frontier.. that's why instead of people getting excited to look for new rtsp superconductor candidates, we get a lot of talk downplaying the only known one. Strong link vs weak link reminds me of how some cultures frown on stimulants while other cultures frown on relaxants.

Re: Replace peer review with “peer replication” (2021)

#70

Post by [deleted] »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

[deleted]
Post reply on HN