Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

wired.com

Post reply on HN
Print view View original

41–50 of 73 posts

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#41

Post by blueMist »

javert wrote: post #39
seccess wrote: post #12

Earlier quoted context omitted.

> Slips through? Mediocre work is the norm, or at least close to being the norm. In computer science, this is likely because of the ridiculous number of conferences there are and how eager each one is to fill out their roster. I want to hear of the conference that published no papers this year because none were good enough for publication. > I don't think it's based on luck, it's based on politics. Academic science i…

> In computer science, this is likely because of the ridiculous number of conferences there are and how eager each one is to fill out their roster. I want to hear of the conference that published no papers this year because none were good enough for publication. Yep. But there is a reason it is this way, which is that to be successul on the academic feeding trough/career path, you have to serve on PC committees and y…

Agree with a lot of what is being said here. Have seen vicious anonymous reviews in CS - as we all have - both due to someone misunderstanding the work but also due to people from "a different camp" simply disliking a given approach. This must be common in other fields as well. Someone said on this thread that anonymity is not a guarantee of quality - completely agree. The move towards "open reviewing" for conferences in CS is very encouraging, however we will see how far it gets.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#42

Post by nemoniac »

What the scientific peer review needs is not anonymity, but accountability. Authors put their reputation on the line. Let reviewers put their reputation on the line too. Good reviews and good reviewers need to be appreciated, not anonymized.

Let valued scientific reviewers gain reputation in a similar way to how contributors to HN, SE and other sites do.

There could even be a viable business model in this. Publishers, be creative!

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#43

Post by ignostic »

What we need here is a more nuanced approach. Anonymity can solve some problems in research, but it will make other worse.

When people are anonymous, they ARE more likely to be truthful in their criticism. They have less incentive to hold back, and it's just human nature to tone down critical feedback when you're critiquing the work of someone with is either influential or an acquaintance. No one likes to make enemies.

On the other hand, anonymity can pretty clearly bring out some of the worst in us. Some people feel little obligation to be fair or honest when their reputation isn't on the line, and so you see people trying to knock down rivals, people they don't like, or random strangers just for the "thrill of the troll."

Imagine if every time you applied for a job your potential employer had access to anonymous feedback on your past work. Some of it might be fair and honest (whether positive or negative), but some of it might be lies from an anonymous coworker with a grudge. Maybe someone is trying to take you down a rung because you got the promotion over them. You could be penalized for any petty reason, and it would stick with you.

Anonymous feedback communicated publicly is much the same. It holds the reviewer and the object of review on unequal footing. Anonymous feedback would be great for an author or even an editor, but it's just not fair to allow the pettiest of people to attack the works of others while wearing a mask. I'd like a system that helps researchers and others invested in the work to solicit anonymous feedback to make the work better. Public-facing commentary, on the other hand, should be tied to an identity.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#44

Post by pc2g4d »

Much of the discussion here seems to have become "anonymity vs. non-anonymity" --- i.e. either anonymity is good for research or it's bad for research. Why not just accept that in some venues there will be anonymity, and in some not, and let them each develop according to the merits of the respective approaches? No need to have all research anonymous or all research clearly attributed to a public identity.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#45

Post by seanmcdirmid »

esbio wrote: post #40

As a person who did research until a few years ago, I must say that the problem is the exact opposite. When you send an article to a journal, the paper gets reviewed by a number of peers, which send their comments back to the Editor on the appropriateness of the claimed work. The problem with this mechanism is that reviewers have no liability, because their comment is anonymous to the author and won't be available to…

> because their comment is anonymous to the author and won't be available to the readers

You can make comments on your papers available; there is nothing against that. I really think, however, peer review comments should actually be published with rejected and accepted papers (at the author's consent, not reviewers) so that conferences can be more transparent.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#46

Post by javert »

blueMist wrote: post #41
javert wrote: post #39

Earlier quoted context omitted.

> In computer science, this is likely because of the ridiculous number of conferences there are and how eager each one is to fill out their roster. I want to hear of the conference that published no papers this year because none were good enough for publication. Yep. But there is a reason it is this way, which is that to be successul on the academic feeding trough/career path, you have to serve on PC committees and y…

Agree with a lot of what is being said here. Have seen vicious anonymous reviews in CS - as we all have - both due to someone misunderstanding the work but also due to people from "a different camp" simply disliking a given approach. This must be common in other fields as well. Someone said on this thread that anonymity is not a guarantee of quality - completely agree. The move towards "open reviewing" for conference…

I'm going to say something controversial.

Minor reform of the referee system is treating the symptoms, not the disease.

The disease is that there is no honest, mutual, voluntary exchange between a party that values a given research project and the party that produces it.

Rather, we have federal bureaucrats handing out money taken from taxpayers willy-nilly via an old-boy network as described above. (The difference between taxpayers and slaves is that taxpayers only must sacrifice a fraction of their productivity to this absurd system, not all of it.)

In CS, all the research we do either goes to benefit shareholders of companies that ultimately profit from it, or (the majority) is just ignored because it's part of the paper mill competition. Those companies should be the ones paying for the research, not the taxpayers. Those companies are free riding, and "we the people" should put an end to it.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#47

Post by kgarten »

I don't get their stance ... peer-review is already "anonymous". There are also a lot of issues with truly anonymous user forums (see 2chan and 4chan). For me it always seems as if the social aspects of communication disappear when one is truly anonymous, e.g. hate speech. The problem is not lack of anonymity but lack of incentives for reviews (I don't get anything from doing a thorough review of a paper and often it's hard to impossible to judge the contribution without dataset and code). It seems peerpub and similar systems will attract people who have the incentive to attack specific authors (as it happened in this case).

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#48

Post by derf_ »

jedbrown wrote: post #18

I take the opposite approach: I sign my reviews and have ever since I was a grad student. It compels me to do a better job reviewing and often leads to further discussion and sometimes collaboration with the authors. Some people don't like what I have to say, but by and large, they respect it. For many topics that I review for, the authors will have a pretty good idea that I wrote the review unless I am intentionally…

Yes, I think anonymity in reviews is largely a myth, even in "double blind" reviews. For many topics, there are only a few research groups actively pursuing them, and you can often tell from what other work they cite, what datasets they use, etc., which lab it is. So identifying at least some of the authors is relatively easy. You might not get the exact grad student right, but you probably know their advisor.

Identifying reviewers is a little harder, but it's usually not difficult to make educated guesses unless they are, as you put it intentionally vague. Otherwise the other comments on this article about bad reviews from 'people from "a different camp" simply disliking a given approach' and the like would be impossible to justify.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#49

Post by colechristensen »

Let's be sure to point out some terrible journalism, reading the following makes me completely disinterested in finishing the article and have considerably less respect for Wired.

> Have you ever questioned the claims that scientists make? For example, last year’s discovery of the so-called “God particle,”

Using the nickname for the Higgs in the context of questioning science claim is nothing but bait for the foolish and misinformed, and in this way anyone scientifically literate should seriously question any claims or opinions in this publication.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#50

Post by Natsu »

tokenadult wrote: post #37
dougmccune wrote: post #21

The actual complaint filed shows pretty nasty behavior. Here's the full complaint: https://retractionwatch.files.wordpress.com/2014/10/filed-co... Someone very clearly was trying to get Dr. Sarkar fired. The PubPeer comments are only one small aspect. But the person then also (I assume anonymously) emailed his new employer to allege fraud, and then went so far as to print out these allegations and stamped them with o…

A complaint is just a pleading before any trial of the facts has occurred. The statements about what may have happened (or may not have happened) are under the heading "FACTUAL ALLEGATIONS" in the document you kindly linked. So to describe the same issues you describe, I would write, "The actual complaint alleges some pretty nasty behavior." We have yet to see how many of those allegations will be believed by a trier…

Not sure why this was downmodded, but I tried to correct that. I do note that people have a bias to believe the first story they hear and it's hard to consciously correct for this. Merely knowing about the bias is not enough to correct it and I realize that I'm not immune to it myself.

As far as I know, I don't know any of the people involved in this, but I've read enough legal complaints to know that they're not always as factual as they first appear, so I try to consciously restrain myself from simply believing it without hearing the rebuttal. Rushing to judgement may be quite common these days, but it's not the sort of behavior fair or intelligent people should engage in.

It's weird that they lead in with the first few comments, which were tame enough that I discount them completely as being an appropriate subject for a lawsuit, though they seem to get progressively worse and some of the later allegations are much more serious. I do wonder why they lead with their weakest cards, though.

That aside, it does seem that the site it exercising editorial control over the comments, which may work against it with respect to the CDA, even though it's trying to do the right thing based on its policy as quoted in the complaint. Given the policy, I wonder how the later comments ever went through, as it seemed like the site wanted to avoid drama like this entirely. Even if this was a lapse of some kind, it might work against them.

As I have not heard the rebuttal, I will refrain from making any judgements one way or another about who is right here. It's only fair, after all.

EDIT: Retraction Watch is covering this case. I believe them to be a reliable, respected and neutral 3rd party: http://retractionwatch.com/2014/12/10/pubpeer-files-motion-d...

Post reply on HN