Live data from Hacker News

Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

wired.com

61–70 of 73 posts

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#61
Something I've noticed is that most online articles about scientific misconduct and invalid results revolve around the medical sciences. Granted, medicine is probably the biggest piece of the scientific pie right now, and other branches of science may have their own problems, but I think it misleads the public to have a title like "scientific peer review is broken," for an article that focuses exclusively on one branch of science.

Disclaimer: I'm a physicist. I'm sure that physics is not beyond critique, but unless somebody is willing to take enough of an interest in the inner workings of physics research to say something specific about it or credibly include it in a generalization, I'd rather say something like "medical peer review is broken."

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#62
A thought suggestion: 1.) why this is important? THE GREAT CHINESE FAMINE - Mao's Hungry Ghosts book. not against mao or china. Belief in Lysenko - aka "we will grow oranges at the North Pole."

will the great famine come again due to BAD SCIENCE to the USA? GMO crops / invasive pests / scientific 'POLITICS.'

2.) Of course will the science review in Japan, the plans are now to DOUBLE THE NUCLEAR POWER plants, after Fukushima 'incident.'

3.) So, as an average citizen of the USA, sometime reading of HN, I volunteer as 'anonymous submitter'.'

4.) I've noticed WORLD SCIENTISTS. that your code base is mostly broken. Even engineering codes use Matlab - What in the blazes??? and compiler tricks.

Obviously, the scientists have NOT heard of Amazon, haskell or even of open source software.

12.) Basic focus folks like me. yes, how many pushups did you do today? even TRIPLE BLIND - see 'scientific method.' That means I as a VOLUNTEER submit the paper. GO AHEAD AND ATTACK 'the paper' of my repuation. I do NOT plan to be a professor at a majolr unversity.

for the record, dropout grad school. and yes, I am still completing my studies when I am a bit older.

Alas, I am not hopeful.

14.) first it was religion, Catholic Church and the inquisition 15.)then it was 'the great leader' like mao, etc or the suddenly the ROBBER BARRONS like Tom Editson (high school dropout) becoming sudden geniuses.

16..) now its, the game let's get the professor fired by going direcctly to the Board of Trustees of California State U or others.

17.) when in doubt use the 'federal agencies' to murky up the waters.

18.) Tobacco is good for you and healthy (when in moderation) some say.

19.)MY LAST SUGGESTION. the paper be chopped into parts.

I don't want to see the strange science or arcane languages. I only want some of the sampled data to SHOOT HOLES THROUGH using python and haskell.

the book is forthcoming - how to lie subtlely with Statistics and meta heuristics... THIS IS THE KEY WEAKNESS OF PEER review.

Who is really a peer? and most are working at the same college and same area.

want proof? See OpenSSH - and apple goto goto fail.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#63

The actual complaint filed shows pretty nasty behavior. Here's the full complaint: https://retractionwatch.files.wordpress.com/2014/10/filed-co... Someone very clearly was trying to get Dr. Sarkar fired. The PubPeer comments are only one small aspect. But the person then also (I assume anonymously) emailed his new employer to allege fraud, and then went so far as to print out these allegations and stamped them with o…

Here's a link to the discussion of one of the related articles on pubpeer https://pubpeer.com/publications/8EB4592F23B61CC3EE7CF29A752...

This is pretty factual discussion, and the fraud allegations are very convincing. Look at the pictures of the gels. (I don't understand why people use Photoshop to create fake images. You could create a fake gel in 30 minutes, make a real photo of it, and noone will ever be able to prove it.)

If I made a discovery like this, I'd also get pissed. I'd probably also contact the employer of this person.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#64
post #6
post #4

Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken plac…

I agree completely. In my experience as a PhD student (comp sci), all conferences make reviewers anonymous. As a result, there is very little accountability regarding the reviews you receive for your work. More than once, I have had papers rejected simply because a single reviewer barely read the paper and dismissed it. These kinds of reviews are very frustrating to receive, not only because they failed to understand…

I've had similar experience, and it can be tough, especially when the lack of interest from a reviewer is obvious. I still joke about the review I got that included the line "English need improving".

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#65
I usually think that peer review is broken, but that is because we almost always miss the point of what the actual purpose of peer review is for. Peer review is NOT for laymen. A scientific paper is part of a much longer process of discovery which includes massive amounts of debate and argument. Papers always reflect they very edge of a field where literally the only people who can judge whether the results are plausible are peers because they are the ones who are also trying to make sense of the same phenomenon. Over time some primary research papers are singled out because they really are excellent explanations of a phenomenon most of the time however you have to go look at a review paper. Furthermore peer review is absolutely NOT about reproducibility, it is a prerequisite that says that it MIGHT be worth trying to reproduce this work or work contingent on these results being valid. There are so many reasons why a result might be wrong that it often takes fields years to figure it out and the published literature is a record of that.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#66
Anonymity is hard in practice when so much of science consist of small niche fields where it can be pretty obvious who the author of a paper is by just reading the content.

A better solution might be to enforce non-anonymous peer-reviews that can be read by the public after the paper has been published.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#67
Not as deep an article as I'd hoped, but it points to a difficult question. As reviewers of scientific papers have unprecedented power over what research gets published, who reviews the reviewers?

(A personal example - I have just spent a week constructing a counterargument to a reviewer who didn't read my paper properly. Imagine having an online discussion where your career success hangs on the response of an anonymous, disengaged flamer to a single post of yours: can you imagine how much effort you put into that post? Sadly this particular argument is pointless and serves nobody; it would be better use of my time to get on with actual research, but that's not how the system works).

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#68
post #66

Anonymity is hard in practice when so much of science consist of small niche fields where it can be pretty obvious who the author of a paper is by just reading the content. A better solution might be to enforce non-anonymous peer-reviews that can be read by the public after the paper has been published.

Some venues are doing this, e.g. f1000. Here's a paper that I reviewed: http://f1000research.com/articles/3-38/v2

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#69
post #40

As a person who did research until a few years ago, I must say that the problem is the exact opposite. When you send an article to a journal, the paper gets reviewed by a number of peers, which send their comments back to the Editor on the appropriateness of the claimed work. The problem with this mechanism is that reviewers have no liability, because their comment is anonymous to the author and won't be available to…

> because their comment is anonymous to the author and won't be available to the readers You can make comments on your papers available; there is nothing against that. I really think, however, peer review comments should actually be published with rejected and accepted papers (at the author's consent, not reviewers) so that conferences can be more transparent.

You can, but 1) there's no official channel 2) nobody will do it if they are not forced.

Re: Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity

#70
post #6
post #4

Anonymity can't fix scientific peer review, it can only replace type I errors with type II errors. Instead of suppressing criticism that ought not to be suppressed, anonymity can (and often does) fail to suppress criticism that really ought to be suppressed because it is in fact false and defamatory. And indeed, what this article is really about is a lawsuit that alleges that this kind of error has in fact taken plac…

I agree completely. In my experience as a PhD student (comp sci), all conferences make reviewers anonymous. As a result, there is very little accountability regarding the reviews you receive for your work. More than once, I have had papers rejected simply because a single reviewer barely read the paper and dismissed it. These kinds of reviews are very frustrating to receive, not only because they failed to understand…

[deleted]
Post reply on HN