Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

321–330 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#321

Post by ebiester »

harimau777 wrote: post #162
ebiester wrote: post #58

Earlier quoted context omitted.

That is my idea... If we opened it up, there's probably more interesting iterations, such as requiring pre-registration for all papers, having papers with pre-registration count as some portion of a full paper even if they fail so long as the pre-registration passed scrutiny, having non-replicated papers count as some portion of a fully replicated paper, and having replication as a separate category such that there i…

Is tenure really as mechanical as "publish this many papers and you get it"? My impression was that it took into account things like impact factor and was much more subjective. If that were the case, then wouldn't you run into problems with whoever decides tenure paying lip service to counting replication or failed pre-registered papers but in practice being biased in favor of original research?

It depends on where you are. The more prestigious the university, the more that impact factors start to matter. There is also the matter of teaching and service, but I'm not talking about the entire tenure system.

Re: Replace peer review with “peer replication” (2021)

#322

Post by ebiester »

tnecniv wrote: post #129
ebiester wrote: post #30

Earlier quoted context omitted.

It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewer papers but x number of replications, and you are expected to have x replications in your specialty.) You also create a grant funding section for replication which is then passed on to these independent systems. (You would have to have some sort of randomizat…

Your proposal has a whole slew of issues. First, people that want to be professors normally do so because they want to steer their research agenda, not repeat what other people are doing without contribution. Second, who works in their lab? Most of the people doing the leg work in a lab are PhD students, and, to graduate, they need to do something novel to write up in their dissertation. Thus, they can’t just replica…

I was thinking that this would be more of a postdoc position, or that most labs would be hybrid: that is, they are doing some portion of original research and replication research. If they are replicating, they are made authors on the paper as a new position on the paper. They get credit, but a different type of credit.

A Ph.D would be expected to perform some replication research as part of their package.

Finally, we would start with the fields that are easy to replicate and move up. We couldn't replicate CERN if we tried. But we could implement this in psychology tomorrow, for example.

Re: Replace peer review with “peer replication” (2021)

#323

Post by oneshtein »

casualscience wrote: post #111
matthewdgreen wrote: post #46

The purpose of science publications is to share new results with other scientists, so others can build on or verify the correctness of the work. There has always been an element of “receiving credit” to this, but the communication aspect is what actually matters from the perspective of maximizing scientific progress. In the distant past, publication was an informal process that mostly involved mailing around letters,…

Most of this is very legit, but this > Still other outsiders, who misunderstand the entire process, are upset that intermediate results are sometimes incorrect. This confuses them, and they’re angry that the process sometimes assigns “points” to people who they perceive as undeserving. So instead of simply accepting that sharing results widely to maximize the chance of verification is the whole point of the publicati…

IMHO, physicist, especially theoretical physicist, must be able to create a physical model of something, to confirm that their mathematical models are somewhat connected to reality. WTF is «wave of probability»? WTF is «bending of space-time»? These things are possible in dream-land physics only.

Re: Replace peer review with “peer replication” (2021)

#324

Post by hooby »

Currently BOTH is being used - peer review is the first pass, reproduction the second.

Peer review might (or might not) weed out a few papers before they ever get to being reproduced - and that a paper "passed" peer review often means very little. (In some journals more, in some less).

You can't replace peer review with peer replication. Reviewers often do volunteer work - supporting their field and the journal by checking submissions just for any grave errors/mistakes. They often spend just 10 to 15 minutes per submission - for hundreds of submissions. It's not realistic to ask those reviewers to do a full replication attempt for hundreds of submissions.

So any attempt to "replace" review with replication, would end up basically removing review altogether, without increasing the amount of replication attempts being made.

Re: Replace peer review with “peer replication” (2021)

#325

Post by hooby »

hooby wrote: post #324

Currently BOTH is being used - peer review is the first pass, reproduction the second. Peer review might (or might not) weed out a few papers before they ever get to being reproduced - and that a paper "passed" peer review often means very little. (In some journals more, in some less). You can't replace peer review with peer replication. Reviewers often do volunteer work - supporting their field and the journal by ch…

I did some work for online journals where papers got published even IF the peer review was bad and rejections were exceptionally rare.

The review score of the abstract was only used to decide on the best topics to invite for a presentation or talk - and the review score of the paper was used to hand out awards, decide "highlighted" papers, and it also influenced how high up in the search results a given paper might appear.

Re: Replace peer review with “peer replication” (2021)

#326

Post by oneshtein »

fastneutron wrote: post #159

As much as I agree with the sentiment, we have to admit it isn't always practical. There's only one LIGO, LHC or JWST, for example. Similarly, not every lab has the resources or know-how to host multi-TB datasets for the general public to pick through, even if they wanted to. I sure didn't when I was a grad student. That said, it infuriates me to no end when I read a Phys. Rev. paper that consists of a computational…

Gravitational waves are confirmed by watching of distant quasars:

https://scitechdaily.com/gravitational-waves-detected-using-...

Hydrodynamic quantum analogs can be uses to study quantum particles at macro-scale:

https://en.wikipedia.org/wiki/Hydrodynamic_quantum_analogs

ESA Euclid near-infrared telescope launched few weeks ago:

https://www.esa.int/Science_Exploration/Space_Science/Euclid...

Re: Replace peer review with “peer replication” (2021)

#327

Post by mike_hearn »

lou1306 wrote: post #311
mike_hearn wrote: post #204

Earlier quoted context omitted.

Is PL theory actually science? Although we call it computer science, I don't personally think CS is actually a science in the sense of studying nature to understand it. Computers are artificial constructs. CS is a lot closer to engineering than science. Indeed it's kind of nonsensical to talk about replicating an experiment in programming language theory. For the "hard" sciences, replication often isn't so difficult…

Well there are entire areas of CS research tangential to PLs (say, SAT/SMT, software verification, program synthesis) where the fundamental problems are known to be NP-complete/exponential/undecidable. So it is pretty hard to declare a new approach "superior" on purely theoretical grounds: you usually have to run some benchmarks and see how your new approach compares to existing alternatives. And you want these bench…

Yes performance involves experiments, but are they scientific experiments? I'm not sure it matters either way, it's a purely semantic debate. The issues that create replicability problems in CS are pretty different to the ones that create issues in other fields. My experience was that they're purely engineering problems rather than problems of incentives or non-replicable designs. If the issue was just that some papers occasionally don't replicate because the authors forgot a detail, they get queried and update the paper then nobody would care about this issue. It gets attention because that's sadly not what happens.

Re: Replace peer review with “peer replication” (2021)

#329

Post by SkyBelow »

matthewdgreen wrote: post #167
dmarchand90 wrote: post #133

Earlier quoted context omitted.

I believe that, contrary to popular belief, the implementation of this system would lead to a substantial increase in productivity in the long run. Here's why: Currently, a significant proportion of research results in various fields cannot be reproduced. This essentially means that a lot of work turns out to be flawed, leading to wasted efforts (you can refer to the 'reproducibility crisis' for more context). Moreov…

In the current system scientists allocate reproduction efforts to results that they intend to build on. So if you’ve claimed a breakthrough technique for levitating widgets — and I think this widget technique can be used to build spacecraft (or if I think your technique is wrong) — then I will allocate precious time and resources to reproducing your work. By contrast if I don’t think your work is significant and wort…

>In the current system scientists allocate reproduction efforts to results that they intend to build on. So if you’ve claimed a breakthrough technique for levitating widgets — and I think this widget technique can be used to build spacecraft (or if I think your technique is wrong) — then I will allocate precious time and resources to reproducing your work.

Wouldn't this imply that worthwile results are already being replicated, so the primary cost has been paid and we just need to give some method to disseminate this work and then have it factor into the credibility of science? pre-print And it'll also show when someone is building on things but not replicating it, which I guess is an issue in some fields more than others.

Re: Replace peer review with “peer replication” (2021)

#330

Post by glial »

ebiester wrote: post #322
tnecniv wrote: post #129

Earlier quoted context omitted.

Your proposal has a whole slew of issues. First, people that want to be professors normally do so because they want to steer their research agenda, not repeat what other people are doing without contribution. Second, who works in their lab? Most of the people doing the leg work in a lab are PhD students, and, to graduate, they need to do something novel to write up in their dissertation. Thus, they can’t just replica…

I was thinking that this would be more of a postdoc position, or that most labs would be hybrid: that is, they are doing some portion of original research and replication research. If they are replicating, they are made authors on the paper as a new position on the paper. They get credit, but a different type of credit. A Ph.D would be expected to perform some replication research as part of their package. Finally, w…

I agree that asking psychology grad students to register & replicate a result as part of their training would be a boon to the field -- and useful experience for them.
Post reply on HN