Publishing: The peer-review scam • Hacker News Board

Re: Publishing: The peer-review scam

#21

Post by jeffbr13 » Thu, Nov 27, 2014, 12:03 AM UTC

jakethedog wrote: post #16

dougmccune wrote: post #2

[reposting from a previous submission that didn't get any traction] If anyone wants a good idea for a startup in the academic journals space, I'd recommend looking into vetting peer reviewers as part of the article submission/review process. There have been a few high profile cases of people faking their peer reviews recently. At Sage we dealt with a big case of that earlier in the year and retracted 60 papers. A new…

The OA EGU Journals have been doing a good job of reviewing. The review process is open to the public with anyone able to read and comment on a paper, while they also invite/require several experts to participate. While some papers are only reviewed by the invited reviewers, many are reviewed by 5+ people. Having the entire process open to the public is the way forward.

This is the correct response for open-access journals.

There's an economic imperative to keep the process secretive in paid-access journals, rather than improving it - they can claim this vetting process is their 'secret sauce' and it's one way many attempt to justify their fees.

As we've learned from open-source, if you can get enough eyes (identity isn't actually that important) then all bugs become shallow.

Re: Publishing: The peer-review scam

#22

Post by ylem » Thu, Nov 27, 2014, 12:41 AM UTC

Wow! I have participated in peer review and the system can work--here, it seems like the editors were lazy. I wonder how many papers a typical editor is responsible for per day? I can understand researchers wanting to exclude groups that they are competing with. I always assumed that editors would check suggested referees against obvious conflicts of interest, but perhaps there really are too many papers and too many groups for editors to manually keep track of this these days? Perhaps some checking for recent coauthorship and institution could be performed? Ultimately, the safest way would be to simply not allow people to suggest referees. Editors should be able to find people in the field (based on the literature) who could potentially referee the paper...

The basic problem is that I think there is value to having referees who are anonymous to the submitter. That way, there is no problem with someone relatively new offering a strong critique of a relatively established researcher. But someone (the editor) needs to know that the referee is competent. Also, competent refereeing takes work. For example, I refereed a "methods" paper awhile back and since the method would likely be encoded into software (and there are a lot of people that don't repeat derivations), I actually went through their integrals, checked for conceptual errors, (-) sign errors, integration errors, etc. It's community service and necessary--but it does take time.

Re: Publishing: The peer-review scam

#23

Post by MCRed » Thu, Nov 27, 2014, 1:32 AM UTC

I'm an author on several physics papers published before I switched careers to become a programmer. So I've been published in "big name" (for physics) journals.

Peer review, even without people scamming it, is still broken. The reason is, presuming you have an ideal peer panel, everyone is competent in the art of the paper. This means that they are all working in schools of research relevant to the paper they are reviewing.

Thus every paper either supports or opposes the peers school. If your paper helps confirm their theory, great. If not, they have a tendency to let politics get involved and reject the paper.

Now, a lot of funding comes from government. Imagine the situation where government is funding research with a desired outcome. This desired outcome has a policy implications. Result: Your peer review system is going to highlight results that confirm the government's policy position because the peers are getting their paycheck there, and it is going to bury results the disagree with the position.

I think that a better system would be to have peers give written reviews, that would be attached to the paper, with a rebuttal from the papers authors. Publish more papers, air differeing views, and make more money (since more paper sold means more library fees, etc.)

Then peer review will be less about "censorship of radical ideas" than about "hypothesis-antithesis-synthesis".

And yes, even in obscure areas of materials physics, there is a lot of politics... cause the stakes are so low.

Re: Publishing: The peer-review scam

#24

Post by privong » Thu, Nov 27, 2014, 1:38 AM UTC

chuckcode wrote: post #11

The current academic peer review system is an example of a process that worked fine when number of participants was small but fails at scale. I'd like to see something like hacker news or reddit's social version of vetting papers in addition to any formal review process, perhaps with additional moderation from known experts. There is currently no site for community review of articles or papers and I think it would be…

For physics and astronomy, there is the arXiv. Essentially all the papers (at least for astronomy) are posted there. The next step would be to add a community voting/vetting scheme on top of that. Statistics could be displayed alongside the article, and access to more detailed comments could be facilitated. Version tracking would enable people to update their papers to address criticism and suggestions. (arXiv already does allow people to replace their papers with new versions, and they retain the old ones, but there isn't really an easy way to "diff" two versions of the paper).

Re: Publishing: The peer-review scam

#25

Post by untilHellbanned » Thu, Nov 27, 2014, 2:05 AM UTC

chuckcode wrote: post #11

The current academic peer review system is an example of a process that worked fine when number of participants was small but fails at scale. I'd like to see something like hacker news or reddit's social version of vetting papers in addition to any formal review process, perhaps with additional moderation from known experts. There is currently no site for community review of articles or papers and I think it would be…

Onarbor, https://onarbor.com, is trying to tackle the scaling issue you mention. Regarding the reviews and reputation building its very much in the model of HN, Reddit, Stackoverflow. There is also a Kickstarter-like funding component. Worth a look.

Re: Publishing: The peer-review scam

#26

Post by untilHellbanned » Thu, Nov 27, 2014, 2:07 AM UTC

protonfish wrote: post #6

ghshephard wrote: post #3

The entire scam seems to be possible because there is no attempt at confirming the credibility of the people presuming to actually do the peer review. This type of decentralized authentication is precisely what PGP is particularly good at. I have to imagine that it would be trivial to design a system which would simultaneously ensure that only the actual scientists would be able to peer review a paper based on other…

If you image this is a trivial problem to solve, I think your imagination is not very good. I'll quote the Bruce Schneier quote from the article. “There are almost never technical solutions to social problems.”

Great quote. It gets me thinking about Facebook. FB is assuredly a massive technical solution to a social problem. I get Bruce's sentiment, but there are exceptions to this rule.

Re: Publishing: The peer-review scam

#27

Post by untilHellbanned » Thu, Nov 27, 2014, 2:08 AM UTC

scottfr wrote: post #8

The fundamental issue with the current peer review system is that most academic research is "valueless" to those involved in the system. I use the term "valueless" in a very careful and specific meaning. It does not mean the work is without merit or doesn't have large impact or benefits to society, rather it simply means the quality of the work has no value to the peer reviewers or to the journal itself. * The peer r…

Onarbor, https://onarbor.com, is doing precisely what you describe. It's publishing and funding. I'm one of its creators so would be grateful if you'd be interested in talking more. tim@onarbor.com.

Re: Publishing: The peer-review scam

#28

Post by untilHellbanned » Thu, Nov 27, 2014, 3:12 AM UTC

scottfr wrote: post #20

GFK_of_xmaspast wrote: post #13

Earlier quoted context omitted.

There is, or should be, more to academia than just the pursuit of money.

Of course. The primary goal of academia is the advancement of human knowledge. Unfortunately, the current ecosystem of journals is to a significant extent failing at that. Possibly introducing a clear [0] profit motive into that system would result in it working more efficiently and better serving academia's primary goal. [0] I say "clear" because a lot of the current system of journals is primarily driven by profit…

The researchers need to profit in order for the incentives to truly align. It's like an incredibly poor man's spotify and we all know how that works for musicians.

Re: Publishing: The peer-review scam

#29

Post by IndianAstronaut » Thu, Nov 27, 2014, 3:58 AM UTC

chuckcode wrote: post #11

The current academic peer review system is an example of a process that worked fine when number of participants was small but fails at scale. I'd like to see something like hacker news or reddit's social version of vetting papers in addition to any formal review process, perhaps with additional moderation from known experts. There is currently no site for community review of articles or papers and I think it would be…

More than just peer review, what is crucial is replication of experiments by many. So often experiments are found too difficult to replicate rendering the results unusable.

Re: Publishing: The peer-review scam

#30

Post by avn2109 » Thu, Nov 27, 2014, 4:04 AM UTC

dougmccune wrote: post #2

[reposting from a previous submission that didn't get any traction] If anyone wants a good idea for a startup in the academic journals space, I'd recommend looking into vetting peer reviewers as part of the article submission/review process. There have been a few high profile cases of people faking their peer reviews recently. At Sage we dealt with a big case of that earlier in the year and retracted 60 papers. A new…

Is there anything stopping the publishers from requiring cryptographically signed reviews by peers? E.g. we establish a bijection between public keys and institutional email addresses and then only accept signed reviews.