Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity
1–10 of 73 posts
Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity
#2The submission here is an interesting article by the founders of PubPeer, which has already been in the news quite a bit for finding examples of shoddy science papers that have had to be withdrawn by journal editors. I learned about PubPeer on the group blog Retraction Watch (RT), and I just bopped over to Retraction Watch after reading the article kindly submitted here. RT reports in detail on the defamation suit against PubPeer that is mentioned in the parent article of this thread.[1] I hope the PubPeer experiment can continue and thrive and promote better scientific research practices.
Some of the other comments here suggest that anonymity of reviewers is dangerous in itself. That's why some researchers promote an open review process. Jelte Wicherts and his co-authors put a set of general suggestions for more open data in science research in an article in Frontiers of Computational Neuroscience (an open-access journal).[2]
"With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses."
Wicherts has published another article, "Publish (Your Data) or (Let the Data) Perish! Why Not Publish Your Data Too?"[3] on how important it is to make data available to other researchers. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence. When I see a new publication of primary research in that discipline, I don't take it seriously at all as a description of the facts of the world until I have read that independent researchers have examined the first author's data and found that they check out. Often the data are unavailable, or were misanalyzed in the first place.
[1] http://retractionwatch.com/2014/12/10/pubpeer-files-motion-d...
[2] Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020
http://www.frontiersin.org/Computational_Neuroscience/10.338...
[3] Wicherts, J.M. & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence,40, 73-76.
Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity
#3The obvious problem here is with fields that are small and highly specialized, such that anyone knowledgeable enough to comment on a topic are known to be from a small circle of scientists.
Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity
#4Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken place.
Ironically, the very title of this article is a model of non-scientific thinking that ought to be subject to criticism, but attempts to inoculate itself against criticism by asserting that the subjects of the piece are protagonists "fighting to fix" a broken system. They're not. They're fighting to replace one broken system with a different broken system.
There is no question that scientific peer review is broken and needs to be fixed. But anonymity is not the answer. And holding anonymity up as something that should itself be beyond criticism is certainly not the answer.
Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity
#5Thank you for helping to promote reproducibility of published results by supporting anonymous peer review! Science Exchange (YC S11) is also making great progress in the facilitation of scientific reproducibility. We just completed independently validating select results from 50 cancer biology papers. https://news.ycombinator.com/item?id=8731274
Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity
#6Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken plac…
I agree completely. In my experience as a PhD student (comp sci), all conferences make reviewers anonymous. As a result, there is very little accountability regarding the reviews you receive for your work. More than once, I have had papers rejected simply because a single reviewer barely read the paper and dismissed it. These kinds of reviews are very frustrating to receive, not only because they failed to understand the basic premises of your paper, but because these reviews contain no useful information on how to make your paper better for future submissions.
Of course, the opposite can happen where mediocre works slips through, and the reviewers that allowed that should be held accountable too. Its painful to me that so much of the acceptance process for research papers (in my field at least) is based on luck.
Moral of the story: reviewer anonymity is good but it comes at the expensive of accountability.
Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity
#7I think the increase in fraudulent papers getting published is a symptom, not the disease. The disease is that there are too many scientists doing research for a given research budget.
Due to the ferocious competition for grant money, people are either sloppy and cut corners, or do whatever they need to do and outright cheat to publish in top tier journals.
As a society, we need to make a decision about how much we want to fund scientific research - then, once we've made that decision, insure that we put in place a sustainable system in place - we cannot put in a put of money to finance 100 grants, but then build a pipeline that funnels an ever increasing amount of people into a pool that remains constant.
Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity
#8Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken plac…
I agree completely. In my experience as a PhD student (comp sci), all conferences make reviewers anonymous. As a result, there is very little accountability regarding the reviews you receive for your work. More than once, I have had papers rejected simply because a single reviewer barely read the paper and dismissed it. These kinds of reviews are very frustrating to receive, not only because they failed to understand…
I'm also a computer science PhD student. I don't disagree with anything you said, but I am even more cynical.
> Of course, the opposite can happen where mediocre works slips through
Slips through? Mediocre work is the norm, or at least close to being the norm. (And, as you say, good work is often rejected.)
> Its painful to me that so much of the acceptance process for research papers (in my field at least) is based on luck.
I don't think it's based on luck, it's based on politics. Academic science is a racket. There is some luck too, though.
Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity
#9Fortunately, the First Amendment is on our side. It protects the right to anonymous speech.
I don't understand how "The government cannot outlaw your speech" means "Your anonymity is protected by law from private parties". How does that interpretation come about?
Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity
#10Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken plac…
I agree completely. In my experience as a PhD student (comp sci), all conferences make reviewers anonymous. As a result, there is very little accountability regarding the reviews you receive for your work. More than once, I have had papers rejected simply because a single reviewer barely read the paper and dismissed it. These kinds of reviews are very frustrating to receive, not only because they failed to understand…
We ended up with 5 peer reviewers and an Editor override for one of our papers, instead of the usual 2 (I was the PhD student who actually did the algebra, so I saw all the comments.)
In fact, even after so much work to get it published, we now know of two errors in it - neither were pointed out by the reviewers of course.
I can't help thinking that that one experience sums up all of your problem cases above :)