Publishing: The peer-review scam • Hacker News Board

Re: Publishing: The peer-review scam

#41

Post by pcrh » Thu, Nov 27, 2014, 12:16 PM UTC

scottfr wrote: post #8

The fundamental issue with the current peer review system is that most academic research is "valueless" to those involved in the system. I use the term "valueless" in a very careful and specific meaning. It does not mean the work is without merit or doesn't have large impact or benefits to society, rather it simply means the quality of the work has no value to the peer reviewers or to the journal itself. * The peer r…

The standards required for patentability are quite different from scientific novelty or interest.

Paying reviewers is not a bad idea, as journals that pay will both be more likely to attract quality reviewers, and it also creates a contract between the reviewer and the journal. Such a contract could include requirements for the standard of reviewing. If a paper is later found to have been poorly reviewed the journal can seek redress from the reviewers, further encouraging them to do their job properly.

Re: Publishing: The peer-review scam

#42

Post by pcrh » Thu, Nov 27, 2014, 12:30 PM UTC

chuckcode wrote: post #11

The current academic peer review system is an example of a process that worked fine when number of participants was small but fails at scale. I'd like to see something like hacker news or reddit's social version of vetting papers in addition to any formal review process, perhaps with additional moderation from known experts. There is currently no site for community review of articles or papers and I think it would be…

Check out pubpeer. It is supposedly post-publication peer review, but in practice is usually limited to people making accusations of data manipulation. The conversations can be a bit more chaotic than is ideal for peer review.

https://pubpeer.com/

Re: Publishing: The peer-review scam

#43

Post by scientist » Thu, Nov 27, 2014, 1:48 PM UTC

chuckcode wrote: post #11

The current academic peer review system is an example of a process that worked fine when number of participants was small but fails at scale. I'd like to see something like hacker news or reddit's social version of vetting papers in addition to any formal review process, perhaps with additional moderation from known experts. There is currently no site for community review of articles or papers and I think it would be…

Epistemio allows rating publications as well as publishing post-publication peer reviews. Even though ratings and reviews may be anonymous, reviewers are authenticated, in order to prevent the kind of issues mentioned in the target article.

http://www.epistemio.com/

Re: Publishing: The peer-review scam

#44

Post by gadders » Thu, Nov 27, 2014, 2:02 PM UTC

MCRed wrote: post #23

I'm an author on several physics papers published before I switched careers to become a programmer. So I've been published in "big name" (for physics) journals. Peer review, even without people scamming it, is still broken. The reason is, presuming you have an ideal peer panel, everyone is competent in the art of the paper. This means that they are all working in schools of research relevant to the paper they are rev…

The above has nothing to do with Global Warming.

Re: Publishing: The peer-review scam

#45

Post by ska » Thu, Nov 27, 2014, 3:36 PM UTC

jeffbr13 wrote: post #21

jakethedog wrote: post #16

Earlier quoted context omitted.

The OA EGU Journals have been doing a good job of reviewing. The review process is open to the public with anyone able to read and comment on a paper, while they also invite/require several experts to participate. While some papers are only reviewed by the invited reviewers, many are reviewed by 5+ people. Having the entire process open to the public is the way forward.

This is the correct response for open-access journals. There's an economic imperative to keep the process secretive in paid-access journals, rather than improving it - they can claim this vetting process is their 'secret sauce' and it's one way many attempt to justify their fees. As we've learned from open-source, if you can get enough eyes (identity isn't actually that important) then all bugs become shallow.

"As we've learned from open-source, if you can get enough eyes (identity isn't actually that important) then all bugs become shallow."

This is an often repeated claim, but I don't think it holds up.

What is more accurate is that, given enough eyes most bugs are shallow, but the deep ones are not.

Some things are just subtle, and some things are technically deep. Throwing a thousand naive[1] reviewers at a problem like that wont get you anywhere near the result of just one person who knows the domain.

In the domain of peer review, this means that you can't replace (domain) expert reviewers with a random sample of interested people. You can certainly enhance the expert reviewers with additional people, and this is a good idea.

[1] Naive with respect to that particular problem domain.