Live data from Hacker News

Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

wired.com

31–40 of 73 posts

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#31
post #26
post #16

Earlier quoted context omitted.

http://en.wikipedia.org/wiki/Dendrite_International,_Inc._v.... . > "the court must balance the defendant's First Amendment right of anonymous free speech against the strength of the prima facie case presented and the necessity for the disclosure of the anonymous defendant's identity."

I don't think that opinion says what you think it says. What is really being discussed there is if the government could compel Yahoo to unmask an anonymous user. It says nothing about actual anonymity guarantees among private parties. If Yahoo decided to hand over the account details, the court would not have intervened at all. So when you have an account with some service, any expectations on how the company will ha…

We know that the government could compel Yahoo to unmask an anonymous user - get a FISA order to inspect the business records and, poof, done. So I don't see how that's the issue. Here's the breakdown:

"A prominent cancer scientist, unhappy with the attention his research papers have received on PubPeer, is suing some of our anonymous commenters for defamation."

That's a third-party, who is not a government organization, suing PubPeer to de-anonyomize a commenter on PubPeer's site.

"Dendrite International, Inc., a purveyor of computer software used in the pharmaceutical industry, brought a John Doe lawsuit against individuals who had anonymously posted criticisms of the company on a Yahoo message board. When Presiding Chancery Judge Kenneth MacKenzie rejected one of Dendrite's requests to compel Yahoo to reveal the identity of an anonymous defendant, Dendrite appealed."

That's a third party, Dendrite, who is not a government organization, suing Yahoo to de-anonyomize a commenter on Yahoo's site.

If Yahoo wanted to, they could turn that information over. The question was, could Dendrite compel Yahoo to do so if Yahoo didn't want to. And the answer is that there's a First Amendment issue which must be considered, so the answer is "sometimes."

Yahoo didn't want to. And PubPeer doesn't want to.

So I don't understand why you are bringing up first party actions (actions that Yahoo and PubPeer are free to do) when the issue is what third party actions may compel them to do.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#32

> A prominent cancer scientist, unhappy with the attention his research papers have received on PubPeer, is suing some of our anonymous commenters for defamation On the other hand, should anonymous commenters have the balance of power: in other words, say whatever they want with impunity, even if it actually is defamatory? (Not saying that is the case in this situation, but in general). The problem is that defamation…

This situation is reminiscent of websites hosting copyrighted material posted by anonymous users. Does the relevant law (DMCA, I think) say anything about defamation? A reasonable compromise might be that websites are held harmless so long as they respond to official defamation complaints in some prescribed manner. (Of course, this might not work as well because defamation is significantly less clear-cut than copyrig…

I think you may mean "prescribe" rather than "proscribe." :-)

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#33
post #30

It seems plausible to imagine a system where PeerPub could retain their "published author to register This model relies on t a blind cryptographic signature.( https://en.wikipedia.org/wiki/Blind_signature ) PeerPub generates a public/private key pair. 1. Alice wishes to register on PeerPub. She generates a Nonce N. 2. She blinds the nonce with a factor B. 3. She submits the blinded nonce and her identity to PeerPub.…

Not specific to PeerPub, but I helped work on a prototype of a federated login system based on OpenID that used blind signatures in a similar fashion:

http://private-idp.appspot.com/

https://code.google.com/p/pseudoid/

http://blind-signer.appspot.com/

I haven't touched it for years and it looks like the login flow is broken now with the demo sites.

Also, this uses unsafe server-side JS crypto for demo purposes only.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#34
post #33
post #30

It seems plausible to imagine a system where PeerPub could retain their "published author to register This model relies on t a blind cryptographic signature.( https://en.wikipedia.org/wiki/Blind_signature ) PeerPub generates a public/private key pair. 1. Alice wishes to register on PeerPub. She generates a Nonce N. 2. She blinds the nonce with a factor B. 3. She submits the blinded nonce and her identity to PeerPub.…

Not specific to PeerPub, but I helped work on a prototype of a federated login system based on OpenID that used blind signatures in a similar fashion: http://private-idp.appspot.com/ https://code.google.com/p/pseudoid/ http://blind-signer.appspot.com/ I haven't touched it for years and it looks like the login flow is broken now with the demo sites. Also, this uses unsafe server-side JS crypto for demo purposes only.

I emailed the PeerPub people about something like this. Will see if they bite.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#35
With Publons.com (https://publons.com) we have different philosophy: the more transparency we can bring to the review process, the better. At the same time we recognise that both blind and double-blind peer review play an important role in generating quality research.

Our approach is to focus on turning review of all kinds (including both pre- and post-publication) into a measurable research output -- something you can add to your resume. We support both anonymous and signed review with the idea that it will lead to greater transparency in the long run and also motivate reviewers to contribute more.

We have a significant number of both types of review now and are starting to look ways to measure if there are significant differences between blind and open review.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#36

Earlier quoted context omitted.

This situation is reminiscent of websites hosting copyrighted material posted by anonymous users. Does the relevant law (DMCA, I think) say anything about defamation? A reasonable compromise might be that websites are held harmless so long as they respond to official defamation complaints in some prescribed manner. (Of course, this might not work as well because defamation is significantly less clear-cut than copyrig…

I think you may mean "prescribe" rather than "proscribe." :-)

Yes, that is an especially confusing error. Thanks.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#37

The actual complaint filed shows pretty nasty behavior. Here's the full complaint: https://retractionwatch.files.wordpress.com/2014/10/filed-co... Someone very clearly was trying to get Dr. Sarkar fired. The PubPeer comments are only one small aspect. But the person then also (I assume anonymously) emailed his new employer to allege fraud, and then went so far as to print out these allegations and stamped them with o…

A complaint is just a pleading before any trial of the facts has occurred. The statements about what may have happened (or may not have happened) are under the heading "FACTUAL ALLEGATIONS" in the document you kindly linked. So to describe the same issues you describe, I would write, "The actual complaint alleges some pretty nasty behavior." We have yet to see how many of those allegations will be believed by a trier of fact at trial.

The plausible evidence of persistent research misconduct in this case seems to be a reasonable basis for someone not to get a new job with more research responsibilities, so there may not be any actual harm to the plaintiff here. The defense of truth is strong in a defamation case.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#38
My understanding is that most peer review systems in place at various journals and funding agencies today are already anonymous (except for when people are identified by their well known viewpoints.) If you ask me, the real problem can't be solved with communications technology, the real problem must be solved at the source: The funding agencies need to take the importance of reproduction of results seriously and require their grantees to do a certain amount of rote reproduction work in order to qualify for grants for novel research. Will it slow the pace of things down? Certainly. Will it increase the quality of the science? Certainly.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#39
post #12
post #8

Earlier quoted context omitted.

I'm also a computer science PhD student. I don't disagree with anything you said, but I am even more cynical. > Of course, the opposite can happen where mediocre works slips through Slips through? Mediocre work is the norm, or at least close to being the norm. (And, as you say, good work is often rejected.) > Its painful to me that so much of the acceptance process for research papers (in my field at least) is based…

> Slips through? Mediocre work is the norm, or at least close to being the norm. In computer science, this is likely because of the ridiculous number of conferences there are and how eager each one is to fill out their roster. I want to hear of the conference that published no papers this year because none were good enough for publication. > I don't think it's based on luck, it's based on politics. Academic science i…

> In computer science, this is likely because of the ridiculous number of conferences there are and how eager each one is to fill out their roster. I want to hear of the conference that published no papers this year because none were good enough for publication.

Yep. But there is a reason it is this way, which is that to be successul on the academic feeding trough/career path, you have to serve on PC committees and you have to publish a lot of papers.

> There is a very real possibility of work being accepted or rejected purely because of what the reviewer thinks will be most beneficial to their career.

I think this is right, but it's less often so work can be scooped, and more often about building up your "camp." As a research professor, you need your area of expertise to be popular and you need to get a lot of citations for your work. You want to be a leader in an area that actually matters, where your definition of "actually matters" is actually purely herd mentality: what do other people consider to be important? So you want to reject work that bolsters your camp, cites you, etc., and reject work that poses a threat to the ascendancy of your area of expertise.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#40
As a person who did research until a few years ago, I must say that the problem is the exact opposite. When you send an article to a journal, the paper gets reviewed by a number of peers, which send their comments back to the Editor on the appropriateness of the claimed work.

The problem with this mechanism is that reviewers have no liability, because their comment is anonymous to the author and won't be available to the readers, as it won't be published as part of the article. The result is that reviewers are not made accountable now or in the future for inaccuracies in their review, blatant attacks, or tactical requests for additional irrelevant investigation just out of spite or to stall you so that they can scoop your paper.

Occasionally, the Editor can step in and disregard a particularly obnoxious reviewer, but it depends on the editor, the journal, and the political/scientific strength of the reviewer.

Post reply on HN