Peer review: the end of an error? • Hacker News Board

Re: Peer review: the end of an error?

#41

Post by kelnage » Wed, Mar 21, 2018, 9:47 PM UTC

aldoushuxley001 wrote: post #25

kelnage wrote: post #2

The initial examples the author uses really do reflect the magnitude of the third problem. All of them were big breakthrough results in their fields - and hence people cared enough to do proper reviews - especially in the latter two cases. As an academic, I have participated in reviews of papers that I would have never otherwise read, but could still contribute to the process. Without something pushing all academics…

"Only interesting problems/solutions/results get published" I'm afraid we're already there. Journals are very much strongly incentivized to publish only interesting or novel results.

I agree there is definitely a bias towards such results - but I fear that completely free-for-all open access reviewing will make the effect even greater.

Re: Peer review: the end of an error?

#42

Post by improbable22 » Wed, Mar 21, 2018, 9:57 PM UTC

aldoushuxley001 wrote: post #25

kelnage wrote: post #2

The initial examples the author uses really do reflect the magnitude of the third problem. All of them were big breakthrough results in their fields - and hence people cared enough to do proper reviews - especially in the latter two cases. As an academic, I have participated in reviews of papers that I would have never otherwise read, but could still contribute to the process. Without something pushing all academics…

"Only interesting problems/solutions/results get published" I'm afraid we're already there. Journals are very much strongly incentivized to publish only interesting or novel results.

Isn't this the point of journals? They communicate new knowledge, which changes the status of what is known.

There's a balance to be struck in how certain these things should be before being published, for sure. The p-hacking plague is a symptom of setting this certainty bar too low, or equivalently, the required level of novelty too high.

Things which stay constant are the job of standards bodies, and textbook writers... these are important too, but different.

Re: Peer review: the end of an error?

#43

Post by femto » Wed, Mar 21, 2018, 9:59 PM UTC

probably_wrong wrote: post #27

There are two points that I wished were addressed more often. 1. New researchers. The examples he mentions include researchers who were already important in the field, and whose prestige guarantees review. But if you're a new researcher, good luck getting noticed. Blind review doesn't have this problem. 2. The cost of journals. People say "journal" and think of Elsevier, but there are several journals and conferences…

One way to look at journals are that they are "ready made" reputation for those without their own. A new author can leverage the reputation of the journal to establish their own reputation.

On that basis, maybe the mark of a truly prestigious researcher is one who can post directly to arXiv? Rather than the highest accolade being for a paper in Nature, the highest accolade should be for a widely reviewed paper on arXiv?

Re: Peer review: the end of an error?

#44

Post by jhbadger » Wed, Mar 21, 2018, 10:04 PM UTC

I think the mathematical sciences see peer review as less of a requirement than the experimental -- in theory, anyone can verify a proof for themselves. But in experimental science you really need someone familiar with the techniques to make sure that they did all the needed controls and didn't do any shortcuts and that's what peer review is useful for.

Re: Peer review: the end of an error?

#45

Post by pmiller2 » Wed, Mar 21, 2018, 10:07 PM UTC

jhbadger wrote: post #44

I think the mathematical sciences see peer review as less of a requirement than the experimental -- in theory, anyone can verify a proof for themselves. But in experimental science you really need someone familiar with the techniques to make sure that they did all the needed controls and didn't do any shortcuts and that's what peer review is useful for.

I disagree. Even with the current peer review system in mathematics, false proofs have made it into the literature. Who knows how many more would creep in if we did away with peer review?

Re: Peer review: the end of an error?

#46

Post by jhbadger » Wed, Mar 21, 2018, 10:08 PM UTC

chrisseaton wrote: post #10

heckless wrote: post #5

Earlier quoted context omitted.

Although reviewership is a useful thing that gets put on CVs, I can't think of a position I've ever seen where "review X papers" a year is part of the job requirement. A more salient point about why it is important to note that peer review is unpaid is that academics, as authors, submit their work---for free---to a journal, where reviewers review it for free, and then the journal turns around and sells the article fo…

This is such an adversarial way to look at it. I see it as a community. We write and share our papers as part of our jobs, we review each others' papers as part of our jobs, we help to run the conference as part of our jobs. Everyone contributes to run the system. Nobody buys articles for $35 a piece. All major universities and companies subscribe to the ACM or whatever your field's organisation is for you. And guess…

The whole reason they charge money is that not everyone who wants to read the papers has access. I work for government cancer lab. We subscribe to a lot of journals. Even so, I see these paywalls come up regularly because we can't subscribe to all journals.

Re: Peer review: the end of an error?

#47

Post by arca_vorago » Wed, Mar 21, 2018, 10:18 PM UTC

dragonwriter wrote: post #40

arca_vorago wrote: post #39

I've said it before and I'll say it again. The focus should be on reproducing studies, not on peer-review. Of course peer review can be of assistance in selecting studies worth putting the effort and funding into reproducing, but peer-review itself is extremely corrupted as a process, but since so many peoples careers depend on it , many of them will not talk about those issues. I used to hold the peer-review and jou…

> The focus should be on reproducing studies, not on peer-review The two aren't separable, since a key part of peer review is assuring that a published study is, in principle, reproducible. You can't focus on reproduction withoit focussing on peer review.

I agree, I just worded it slightly poorly. My point is that the biggest focus should be on reproducing studies, not that peer-review shouldn't be revamped also.

I mean, isn't reproducing a study the ultimate peer-review?

Re: Peer review: the end of an error?

#48

Post by noobermin » Wed, Mar 21, 2018, 10:19 PM UTC

jhbadger wrote: post #44

I think the mathematical sciences see peer review as less of a requirement than the experimental -- in theory, anyone can verify a proof for themselves. But in experimental science you really need someone familiar with the techniques to make sure that they did all the needed controls and didn't do any shortcuts and that's what peer review is useful for.

Was going to comment to say the exact same thing. OP used a paper by medical professionals to poo-poo formal peer review and then used math papers as examples of successful arxiv papers scrutinized by other scientists. After realizing that, I stopped reading the article then Googled his name, and--surprise, surprise--he's a mathematician.

It will work for mathy and theoretical sciences like CS and theoretical physics, but good luck getting experimentalists to review papers for free (or at least without being asked by a journal). I'm not sure whether peer review is necessary in those cases, but the culture around more applied sciences definitely considers peer review more important, even though we routinely post to arxiv anyway.

Re: Peer review: the end of an error?

#49

Post by noobermin » Wed, Mar 21, 2018, 10:28 PM UTC

improbable22 wrote: post #16

A point not mentioned in the article is that this formal peer review system is itself quite young. I think that sending every paper out to several reviewers only became the default once we had xerox machines -- 1960s? We did a lot of serious science before that. A bit earlier, the famous anecdote is about Einstein's offence that the Phys Rev editor would dare to send his paper for review at all -- apparently this wou…

If anything, that works as a great defense of peer review given Einstein was wrong.

Re: Peer review: the end of an error?

#50

Post by pizlonator » Wed, Mar 21, 2018, 10:39 PM UTC

I've published in CS conferences, CS journals, and on a blog.

Publishing in CS conferences/journals means you get reviews from 3-6 "smart people", and then maybe 50-100 people will read your paper, and most of those people will never contact you.

OTOH if you publish a blog and advertise on hackernews and twitter, you get instant feedback, including from some of the same experts that would have done reviews for conferences/journals.