Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

261–270 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#261

Post by titzer »

ramesh31 wrote: post #230
titzer wrote: post #86

In the PL field, conferences have started to allow authors to submit packaged artifacts (typically, source code, input data, training data, etc) that are evaluated separately, typically post-review. The artifacts are evaluated by a separate committee, usually graduate students. As usual, everything is volunteer. Even with explicit instructions, it is hard enough to even get the same code to run in a different environ…

> Even with explicit instructions, it is hard enough to even get the same code to run in a different environment and give the same results. Is it really that hard for researchers to standardize around providing Dockerfiles? Environment replication is a solved problem.

> Environment replication is a solved problem.

Unfortunately, no. Dockerfiles aren't as portable as you think, and not architecture-independent. VMs are better, but even then, performance isn't portable either.

The last artifact I produced included builds of 3 web browsers from source--it was over 10GB. One doesn't just "build Chrome in a dockerfile".

Re: Replace peer review with “peer replication” (2021)

#262

Post by andromaton »

Nevermark wrote: post #163

Reproducibility would become a much higher priority if electronic versions of papers are required (by their distributors, archives, institutions, ...) to have reproduction sections, which the authors are encouraged to update over time. UPDATABLE COVER PAGE: Title Authors Abstract Blah, blah, ... State of reproduction: Not reproduced. Successful reproductions: ...citations... Reproduction attempts: ...citations... Cou…

I think Tim Berners-Lee would approve.

Re: Replace peer review with “peer replication” (2021)

#264

Post by faeriechangling »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

In some fields research can’t be replicated later. Much of all autism research will NEVER be replicated because the population of those considered autistic is not stable over time.

Other research proves impossible to replicate because whatever experiment was not described in enough detail to actually replicate it, which should be grounds to immediately dismiss the research before publishing, but which can’t truly be caught if you don’t actually try to reproduce.

Finally these practical concerns don’t even touch on the biggest benefit of reproduction as standard which is that almost nobody wants to reproduce research as they are not rewarded for doing so. This would give somebody, namely those who want to publish something, a strong impetus to get that reproduction done which wouldn’t otherwise exist.

Re: Replace peer review with “peer replication” (2021)

#265

Post by 10g1k »

Peer review is also not part of the scientific method. It's nice, but it's not strictly part of the method.

It may be more accurate to suggest that repeatability is part of the scientific method. But even that is not strictly true.

Consider, the single longest running scientific work was not repeatable, and was not shared with anyone outside the cadre of people doing it. Around 3000 years ago, a secretive caste of astrologers/scribes watched the heavens, and recorded their observations for several centuries. They did not publish their findings, thus making them anecdotal (yes, that's what anecdotal means, just that it wasn't published). The exact circumstances and variables were never repeatable, due to the movements of the celestial bodies, precession, etc.

Similarly, the UQ pitch drop experiment, having not yet completed, has not been repeated. But it's still an entirely valid scientific experiment.

Re: Replace peer review with “peer replication” (2021)

#266

Post by TrackerFF »

peteradio wrote: post #235
jonnycomputer wrote: post #233

If they are, in fact, implying that another lab should produce a matching data-set to try to replicate results, well, I'm sorry, but that won't work, at least in a whole lot of fields. Data collection can be very expensive, and take a lot of time. It certainly is in my field. If on, the other hand, they just want the raw data, and let others go to town on it in their own way, that's fine, probably. Results that don't…

What field is it too expensive or difficult to reproduce the data?

On the top of my head: Say you want to research biological data from the Mariana trench.

Or even worse: Some passing by comet, or planet/moon/whatever in the solar system. And just to make things EVEN worse, you need to analyze the data in some destructive way.

Certainly very plausible scenarios, but also some which could prohibitively expensive to do multiple times.

Re: Replace peer review with “peer replication” (2021)

#267

Post by faeriechangling »

cycomanic wrote: post #174
orangepurple wrote: post #128

Earlier quoted context omitted.

Terrible analogy. It might take months to come up with an idea but another should be able to follow your method and implement it much more quickly than it took you to come up with the concept and implement it.

I think you don't understand how much work is involved in just building the techniques and expertise to pull some experiments off (let's not even talk about the equipment). Even if someone meticulously documents their process, it could still take months to replicate the results. I'm familiar with lithography/nanofabrication and I know that it is typically the case that a process developed in one clean-room can not be…

You are making an argument for replication not against it by stressing how meticulously documenting your process is an “if”, strewing how things like a supplier for a chemical can change and render reproduction impossible, and even if an observation can only be replicated in one clean room that means you effectively only have as long as that clean room remains opens to replicate it.

You are almost stressing all the ways we are producing garbage rendered non-reproducible with deficient documentation of processes, changes in supply, and changes in the environment. All three can be minimized through peer replication.

Re: Replace peer review with “peer replication” (2021)

#268

Post by jxramos »

kergonath wrote: post #19
miga wrote: post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

Barriers to publication should be lower for replication studies, I think that’s the main problem. If someone wants to spend some time replicating something that’s only been described in a paper or two, that is valuable work for the community and should be encouraged. If the person is a PhD student using that as an opportunity to hone their skills, it’s even better. It’s not glamorous, it’s not something entirely new,…

I wonder if undergrads could be harnessed to enter into this kind of work, maybe under the supervision of doctoral students and a well meaning and interested PI.

Re: Replace peer review with “peer replication” (2021)

#269

Post by [deleted] »

p1esk wrote: post #124
omgwtfbyobbq wrote: post #89

Earlier quoted context omitted.

What about a system where peer replication is required if the number of citations exceeds some threshold?

Who will be replicating it? Why would I want to set aside my own research to replicate some claim someone made? How would this help my career?

[deleted]

Re: Replace peer review with “peer replication” (2021)

#270

Post by kibibu »

jakear wrote: post #252
kibibu wrote: post #246

Earlier quoted context omitted.

There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv. One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of thei…

Industry doesn't need peer review. The proof of the pudding is in the sales.

Then why does it dress its marketing up in academic paper format?
Post reply on HN