Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

281–290 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#281

Post by kjkjadksj »

okaleniuk wrote: post #275

Back when I was active in academia, our publishers were reluctant to print source code or even repository links (that was largely before GitHub) but they could still share a paper source on demand. If you reference someone else's paper and want to quote some formula, it is easier and less error prone to copy rather than retype. At that point I thought about making a TeX interpreter so one could easily "run a paper" o…

“Running a paper” is honestly challenging these days because of the resource requirements of a lot of scientific code, or the size of certain datasets. One group might have access to a beefy cluster and there’s no pressure for very performant code when they can parallelize the work across a few dozen xeons or have access to tbs of memory. Another group might be running their code on a laptop. Maybe if your data is much larger than the authors data, their code doesn’t even work since it was designed for much smaller datasets.

Tools like nextflow or snakemake help with respect to having a one liner to generate all data in a paper potentially, handle dependencies, list resource expectations, use your own profile to handle your environment specific job scheduling commands and parameters. However, this still doesn’t do anything for whether you have access to the resources needed.

Re: Replace peer review with “peer replication” (2021)

#282

Post by aeternum »

techdragon wrote: post #98
kergonath wrote: post #50

Earlier quoted context omitted.

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. They also tend to over-estimate the effect of peer review (often equating peer review with validity). > If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort…

This waste of effort by way of duplicating unpublished negative results is a big factor in why replicated results deserve to be rated more highly than results that have not been replicated regardless of the prestige of the researchers or the institutions involved… if no one can prove your work work was correct… how much can anyone trust your work… I have gone down the rabbit hole of engineering research before and 90…

Publishing positive/noteworth results only does seem like an embarrassingly obvious major flaw in academia and the greater scientific community.

A research assistant would quickly be thrown out if he/she refused to record negative experimental results, yet we somehow decide that is fine when operating as a collective.

Re: Replace peer review with “peer replication” (2021)

#283

Post by kjkjadksj »

NalNezumi wrote: post #6
miga wrote: post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

Isn't readability and comprehensibility the job of the editor/journal to check. (after all they're actually paid) maybe not for conference, but peer review is more for checking if the methodology, scope, claim, direction, conclusion and relevances is sound&trustable. At least that's my understanding

The editor is basically deferring to people with expertise who can put the paper into context better than they could. The editor might be an expert in the field, but they can’ speak for every aspect of it like someone working day to day in that specific aspect of the field could. Sometimes the authors themselves even recommend potentially relevant reviewers for the editor to contact for peer reviewing.

Re: Replace peer review with “peer replication” (2021)

#284

Post by snowwrestler »

mike_hearn wrote: post #191
matthewdgreen wrote: post #60

Earlier quoted context omitted.

No subset of humanity is “pure hearted.” Fraud and malice will exist in everything people do. Fortunately these fraudulent incidents seem relatively rare, when one compares the number of reported incidents to the number of publications and scientists. But this doesn’t change anything. The benefit of scientific publication is to make it easier to detect and verify incorrect results , which is exactly what happened in…

Fraud isn't exceedingly rare :( It only seems that way because academia doesn't pay anyone to find it, reacts to volunteer reports by ignoring it, and the media generally isn't interested. Fraud is so frequent and easy to find that there are volunteers who in their spare time manage to routinely uncover not just individual instances of fraud but entire companies whose sole purpose is to generate and sell fake papers…

I hope you appreciate the futility of trying to prove that fraud is ruining science publication by linking to a bunch of publications that capably detect and point out all the fraud.

Re: Replace peer review with “peer replication” (2021)

#285

Post by snowwrestler »

NalNezumi wrote: post #6
miga wrote: post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

Isn't readability and comprehensibility the job of the editor/journal to check. (after all they're actually paid) maybe not for conference, but peer review is more for checking if the methodology, scope, claim, direction, conclusion and relevances is sound&trustable. At least that's my understanding

> Isn’t readability and comprehensibility the job of the editor/journal to check

Yes, who do you think ask the reviewers to perform their reviews?

> peer review is more for checking if the methodology, scope, claim, direction, conclusion and relevances is sound&trustable.

No, the parent comment has it right. The only thing being reviewed is the paper, and the point is to make sure it communicates clearly, not that it’s “sound and trustable.”

Re: Replace peer review with “peer replication” (2021)

#286

Post by snowwrestler »

janalsncm wrote: post #94
analog31 wrote: post #78

Earlier quoted context omitted.

Every experimental paper I've ever read has contained an "Experimental" section, where they provide the details on how they did it. Those sections tend to be general enough, albeit concise. In some fields, aside from specialized knowledge, good experimental work requires what we call "hands." For instance, handling air sensitive compounds, or anything in a condensed or crystalline state. In my thesis experiment, some…

“Concise” isn’t good enough. If other scientists are trying to read through the tea leaves at what you’re trying to say you did, that defeats the entire point of a paper. The purpose of science is to create knowledge that other people can use and if people can’t replicate your work that’s not science.

Yes, maybe there should be a process where peer scientists review a paper before it’s published, to make sure it is written clearly enough for other scientists to understand it.

Re: Replace peer review with “peer replication” (2021)

#287

Post by vkou »

Knee_Pain wrote: post #278
p1esk wrote: post #160

Earlier quoted context omitted.

Because I want to do original research, and be known for doing original research. Only if I fail at that, I might settle for being a guy who reproduces others’ work (which basically means the transition from a researcher to an engineer).

So you want to be known for original results, which cannot be confirmed to be true? So what kind of results are they? Also your mentality is exactly part of the problem: you arrogantly believe that replication work is beneath you and that originality is all that matters.

If what the parent poster discovers is interesting, and other people consider it to be valuable, they will replicate their results in the process of building on top of it.

High-impact work that people care about gets built on, and replicated. Low impact work does not.

Re: Replace peer review with “peer replication” (2021)

#288

Post by tsimionescu »

jakear wrote: post #258
ModernMech wrote: post #257

Earlier quoted context omitted.

This is the final stage of capitalism, where market success is conflated with scientific rigor.

Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.

A quick trip around audiophile companies should quickly disabuse you of the notion that high sales implies any kind of worth to a product. There are several companies making lots of money on selling gold HDMI, ethernet etc cables "to improve sound quality", and plenty of rubes buying them and "hearing the difference".

Re: Replace peer review with “peer replication” (2021)

#289

Post by fodkodrasz »

abnry wrote: post #166
fodkodrasz wrote: post #65

Earlier quoted context omitted.

I guess if software developers will complain that it's too hard or too expensive to thoroughly test their code to ensure exactly zero bugs at release[1], then that just shows their work is BS. [1]: if you have delivered telco code to Softbank you may have heard this sentence

Replication is not the same thing as zero bugs in software.

https://www.merriam-webster.com/dictionary/analogy

Re: Replace peer review with “peer replication” (2021)

#290

Post by memefrog »

tsimionescu wrote: post #288
jakear wrote: post #258

Earlier quoted context omitted.

Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.

A quick trip around audiophile companies should quickly disabuse you of the notion that high sales implies any kind of worth to a product. There are several companies making lots of money on selling gold HDMI, ethernet etc cables "to improve sound quality", and plenty of rubes buying them and "hearing the difference".

Real Hi-Fi companies like Sennheiser and Bose make far more money than people selling gold HDMI cables. And they've been making much more money for decades and will continue to make money for decades.

Grifters don't win in the long-term.

Post reply on HN