Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

wired.com

Post reply on HN
Print view View original

11–20 of 73 posts

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#11

Post by ejz »

One of the main issues is that this is just a really tedious process. No one wants to go through algebra; that's why it's siphoned off to grad students. Do you really think a tenured professor is going to spend time checking the grunt work on /someone else's/ paper when they won't do it for their own? I'm intrigued by the possibility of using natural language processing and logical system tools like Wolfram Alpha. Wolfram recently posted on his blog about building machines that could store data about complex mathematical objects, and already you can build machines that confirm first order logical statements.

Farming out low-level tasks to automated systems would be interesting. Imagine if the format of papers changed entirely, ie, you had to submit your proofs in certain formats, or at least certain parts in specific ways. I'm sure that many professors would be elated to see the number of papers they have to review go down drastically; although, I'm sure many will be disappointed to get a return letter that says, "I'm sorry, but the low level flaws were so serious that they were automatically rejected and are not fit for review."

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#12

Post by seccess »

javert wrote: post #8
seccess wrote: post #6

Earlier quoted context omitted.

I agree completely. In my experience as a PhD student (comp sci), all conferences make reviewers anonymous. As a result, there is very little accountability regarding the reviews you receive for your work. More than once, I have had papers rejected simply because a single reviewer barely read the paper and dismissed it. These kinds of reviews are very frustrating to receive, not only because they failed to understand…

I'm also a computer science PhD student. I don't disagree with anything you said, but I am even more cynical. > Of course, the opposite can happen where mediocre works slips through Slips through? Mediocre work is the norm, or at least close to being the norm. (And, as you say, good work is often rejected.) > Its painful to me that so much of the acceptance process for research papers (in my field at least) is based…

> Slips through? Mediocre work is the norm, or at least close to being the norm.

In computer science, this is likely because of the ridiculous number of conferences there are and how eager each one is to fill out their roster. I want to hear of the conference that published no papers this year because none were good enough for publication.

> I don't think it's based on luck, it's based on politics. Academic science is a racket.

I've heard (more than once) of people who submit a paper, have it rejected, and then get scooped by a professor who was on the PC of the conference that rejected them. There is a very real possibility of work being accepted or rejected purely because of what the reviewer thinks will be most beneficial to their career.

So, I think your cynicism is well-placed.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#13

Post by dluan »

Fede_V wrote: post #7

I think the increase in fraudulent papers getting published is a symptom, not the disease. The disease is that there are too many scientists doing research for a given research budget. Due to the ferocious competition for grant money, people are either sloppy and cut corners, or do whatever they need to do and outright cheat to publish in top tier journals. As a society, we need to make a decision about how much we w…

Funding review is different than pre-publication or even post-publication review of results.

One argument is that review of results is easier for a reviewee to defend in public, since the data and results are stated and there's nothing to hide. Reviewing funding proposals is more speculative once you get past basic methods or ethics review (e.g. depends on the risk appetite of the funders) and therefore might require anonymity.

This is something we've been grappling at Experiment. It's possible a better design of the system is a double blind review for proposals but a public post-pub review for results.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#14

Post by Animats »

Another anonymous review system. What could possibly go wrong?

If there's anything we've learned by now about "crowdsourced" review systems, it's that, without an elaborate way to evaluate reviewers, they fail. Badly. Facebook "likes", Google "+1", and Yelp reviews are heavily spammed. This just does not work.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#15

Post by dluan »

lisper wrote: post #4

Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken plac…

Debatable on whether or not you can assume litigation is grounds for defamation. Even if it is true, it's not necessary that anonymity required the scientist to sue.

I think the appeal for the case to drop is not to discourage discussion of anonymity in review, but that it isn't discussion. A jury panel that decides to outcome of a paper could make science worse off than through a panel of peers.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#16

Post by dalke »

vacri wrote: post #9

Fortunately, the First Amendment is on our side. It protects the right to anonymous speech. I don't understand how "The government cannot outlaw your speech" means "Your anonymity is protected by law from private parties". How does that interpretation come about?

http://en.wikipedia.org/wiki/Dendrite_International,_Inc._v.... .

> "the court must balance the defendant's First Amendment right of anonymous free speech against the strength of the prima facie case presented and the necessity for the disclosure of the anonymous defendant's identity."

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#17

Post by lisper »

dluan wrote: post #15
lisper wrote: post #4

Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken plac…

Debatable on whether or not you can assume litigation is grounds for defamation. Even if it is true, it's not necessary that anonymity required the scientist to sue. I think the appeal for the case to drop is not to discourage discussion of anonymity in review, but that it isn't discussion. A jury panel that decides to outcome of a paper could make science worse off than through a panel of peers.

The suit it a red herring. The point is that anonymity makes it more likely that bogus criticism will be heard, just as non-anonymity makes it more likely that valid criticism will not be heard. Anonymity is not a quality filter.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#18

Post by jedbrown »

I take the opposite approach: I sign my reviews and have ever since I was a grad student. It compels me to do a better job reviewing and often leads to further discussion and sometimes collaboration with the authors. Some people don't like what I have to say, but by and large, they respect it. For many topics that I review for, the authors will have a pretty good idea that I wrote the review unless I am intentionally vague. I would rather write clearly and directly and stand behind it. It is a professional risk that I don't think anyone should be compelled to take, but I think signing reviews is generally good for science.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#19

Post by ylem »

ejz wrote: post #11

One of the main issues is that this is just a really tedious process. No one wants to go through algebra; that's why it's siphoned off to grad students. Do you really think a tenured professor is going to spend time checking the grunt work on /someone else's/ paper when they won't do it for their own? I'm intrigued by the possibility of using natural language processing and logical system tools like Wolfram Alpha. Wo…

I'm a permanent staff member and I do actually check "algebra" and integrals when reviewing papers. Especially "technique" papers, because after the initial paper, people tend to make black boxes, so if the initial errors aren't caught then they can propagate for quite some time...I view it as community service.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#20

Post by snowballsteve »

Fede_V wrote: post #7

I think the increase in fraudulent papers getting published is a symptom, not the disease. The disease is that there are too many scientists doing research for a given research budget. Due to the ferocious competition for grant money, people are either sloppy and cut corners, or do whatever they need to do and outright cheat to publish in top tier journals. As a society, we need to make a decision about how much we w…

One could argue that papers are the disease. It was a good medium, but with today's communication technology, something better needs to arise. The need to "publish to exist" in academia has gotten to the point that it promotes sub-standard work and hinders advancement of knowledge.
Post reply on HN