Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

251–260 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#251

Post by GravelRocks »

matthewdgreen wrote: post #167
dmarchand90 wrote: post #133

Earlier quoted context omitted.

I believe that, contrary to popular belief, the implementation of this system would lead to a substantial increase in productivity in the long run. Here's why: Currently, a significant proportion of research results in various fields cannot be reproduced. This essentially means that a lot of work turns out to be flawed, leading to wasted efforts (you can refer to the 'reproducibility crisis' for more context). Moreov…

In the current system scientists allocate reproduction efforts to results that they intend to build on. So if you’ve claimed a breakthrough technique for levitating widgets — and I think this widget technique can be used to build spacecraft (or if I think your technique is wrong) — then I will allocate precious time and resources to reproducing your work. By contrast if I don’t think your work is significant and wort…

"Cat fart" research might also be incredibly expensive to replicate compared to the "might cure cancer" research. In that case it would effectively get a bigger slice of resources because we're treating all research the same!

Re: Replace peer review with “peer replication” (2021)

#252

Post by jakear »

kibibu wrote: post #246
jakear wrote: post #172

Earlier quoted context omitted.

It's the general public that equates "peer reviewed" with "definitely correct, does not need to be questioned".

There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv. One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of thei…

Industry doesn't need peer review. The proof of the pudding is in the sales.

Re: Replace peer review with “peer replication” (2021)

#253

Post by awesomeMilou »

RugnirViking wrote: post #179
fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

lets be brutally honest with ourselves. 99% of all papers mean nothing. They add nothing to the collective knowledge of humanity. In my field of robotics there are SOOO many papers that are basically taking three or four established algorithms/machine learning models, and applying them to off-the-shelf hardware. The kind of thing any person educated in the field could almost guess the results exactly. Hundreds of suc…

Thank you. Not even saying this to shit on academia, but modern scientific publishing follows the same governing rules as publishing a YouTube video (in principle).

> There should probably be some sort of separate process for things that actually claim to make important discoveries.

This used to be Springer Nature and the likes, but they've had so many retractions in the past years + they broke their integrity in the Schoen scandal, allowing lenience in the review process to secure a prestigious publication in their journal.

In reality, I mean you're probably my academic senior: How does true advancement get publicized these days? You post a YouTube video somewhere. See LK99. No peer review, no fancy stuff, a YouTube video was enough to get Argonne National lab on the case.

Re: Replace peer review with “peer replication” (2021)

#254

Post by jonnycomputer »

peteradio wrote: post #235
jonnycomputer wrote: post #233

If they are, in fact, implying that another lab should produce a matching data-set to try to replicate results, well, I'm sorry, but that won't work, at least in a whole lot of fields. Data collection can be very expensive, and take a lot of time. It certainly is in my field. If on, the other hand, they just want the raw data, and let others go to town on it in their own way, that's fine, probably. Results that don't…

What field is it too expensive or difficult to reproduce the data?

Lots?

Human subjects research, for one e.g. very often they involve clinical populations that are very hard to recruit. You can spend tens of thousands in advertising, and multiples more in labor, to get a hundred participants in, over the course of an entire year of effort, and that's not even counting the money spent on a clinician doing a diagnosis. And then, when you do, you may, say, pay $1000 for the MRI per subject, plus the $100 bucks you pay directly to the participant themselves.

Re: Replace peer review with “peer replication” (2021)

#255

Post by [deleted] »

peteradio wrote: post #235
jonnycomputer wrote: post #233

If they are, in fact, implying that another lab should produce a matching data-set to try to replicate results, well, I'm sorry, but that won't work, at least in a whole lot of fields. Data collection can be very expensive, and take a lot of time. It certainly is in my field. If on, the other hand, they just want the raw data, and let others go to town on it in their own way, that's fine, probably. Results that don't…

What field is it too expensive or difficult to reproduce the data?

[deleted]

Re: Replace peer review with “peer replication” (2021)

#256

Post by jxramos »

You know what I would love to see is metadata attributes surrounding a paper such as [retracted], [reproduced], [rejected], etc. We already have the preprint thing down. Some of these would be implied by being published, ie not a preprint. Maybe even a quick symbol for what method of proof was relied upon—-video evidence, randomized control trial, observational study, Sample count of n>1000 (predefined inequality brackets), etc. I think having this quick digest of information would help an individual wade through a lot of studies quickly.

Re: Replace peer review with “peer replication” (2021)

#257

Post by ModernMech »

jakear wrote: post #252
kibibu wrote: post #246

Earlier quoted context omitted.

There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv. One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of thei…

Industry doesn't need peer review. The proof of the pudding is in the sales.

This is the final stage of capitalism, where market success is conflated with scientific rigor.

Re: Replace peer review with “peer replication” (2021)

#258

Post by jakear »

ModernMech wrote: post #257
jakear wrote: post #252

Earlier quoted context omitted.

Industry doesn't need peer review. The proof of the pudding is in the sales.

This is the final stage of capitalism, where market success is conflated with scientific rigor.

Engineering, not scientific.

High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it.

Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.

Re: Replace peer review with “peer replication” (2021)

#259

Post by jxramos »

NalNezumi wrote: post #6
miga wrote: post #4

Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.

Isn't readability and comprehensibility the job of the editor/journal to check. (after all they're actually paid) maybe not for conference, but peer review is more for checking if the methodology, scope, claim, direction, conclusion and relevances is sound&trustable. At least that's my understanding

Yes a metadata relationship link would be outstanding. Reproduced in some paper xyz, or by some institution, named individuals, etc. some kind of structured information would be very useful.
Post reply on HN