Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Authors’ names have ‘astonishing’ influence on peer reviewers: study

nature.com

Post reply on HN
Print view View original

231–240 of 319 posts

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#231

Post by PeterisP »

jedberg wrote: post #137
hannob wrote: post #134

Earlier quoted context omitted.

> It blows my mind that peer reviews aren't done blind. I've co-authored a few papers and blind reviews have some surprising consequences, like discussions along the lines of "you can't say that you already wrote about this issue elsewhere and put a link in the paper, because that would unblind it". I was a bit uncomfortable with that, because I like to "cite my sources" (even if the source was myself in this case).…

Whoever told you that about your cites is probably right -- you shouldn't cite yourself while calling out that you are citing yourself. You should cite yourself the same you that site any other work. The second part is probably unavoidable. If you're working on something super specialized when all your reviewers are the six other people who work on it then sure, it'll be unblinded. Not much we can do there sadly.

In some cases the issue that you can't really "blind" it because it is a physical continuation of the earlier work - not just building upon the idea, which anyone could do, but using the exact same experimental device (a unique one, like the hadron collider) or continued analysis of the exact same set of patients or animals, or improving the previously made software tool.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#232

Post by jonathanstrange »

rusticpenn wrote: post #221
jonathanstrange wrote: post #213

Earlier quoted context omitted.

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

You can easily find the authors based on the references they use and the topic.

Sure but the idea is that you don't try to find out who the author is, and if you recognize the author anyway, you decline the review. I review regularly and that's what I do whenever I recognize an author.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#233

Post by jsemrau »

This is known. Hence, when we were building the peer-reviewed moderation / curation function for finclout we only show the content. While in selected cases the poster is identifiable, for most users the decision is done only based on content.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#234

Post by kalimanzaro »

jonathanstrange wrote: post #213
epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

What is your discipline? I'm in physics and double blind peer review is not the norm.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#235

Post by joxel »

jedberg wrote: post #137
hannob wrote: post #134

Earlier quoted context omitted.

> It blows my mind that peer reviews aren't done blind. I've co-authored a few papers and blind reviews have some surprising consequences, like discussions along the lines of "you can't say that you already wrote about this issue elsewhere and put a link in the paper, because that would unblind it". I was a bit uncomfortable with that, because I like to "cite my sources" (even if the source was myself in this case).…

Whoever told you that about your cites is probably right -- you shouldn't cite yourself while calling out that you are citing yourself. You should cite yourself the same you that site any other work. The second part is probably unavoidable. If you're working on something super specialized when all your reviewers are the six other people who work on it then sure, it'll be unblinded. Not much we can do there sadly.

This happens in larger fields too. For example in AI, which is a large research field, you can be blind-reviewing a paper, but it uses a proprietary google dataset, so you know where it comes from. Blind review is not the answer, I fear.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#236

Post by kalimanzaro »

epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I suspect the mechanism goes something like this:

Reviewers prioritize their time above all, and after that correctness，leaving novelty in last place. If a paper comes from a unknown lab it tends to get greater scrutiny because a famous name is a subconscious stand-in for correctness. You spend less time reviewing famous authors because you think they are less likely to have bugs.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#237

Post by pbhjpbhj »

jonathanstrange wrote: post #232
rusticpenn wrote: post #221

Earlier quoted context omitted.

You can easily find the authors based on the references they use and the topic.

Sure but the idea is that you don't try to find out who the author is, and if you recognize the author anyway, you decline the review. I review regularly and that's what I do whenever I recognize an author.

Don't you recognise authors on topics you're most competent to review on?

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#238

Post by jonathanstrange »

Orothrim wrote: post #229
techas wrote: post #220

Earlier quoted context omitted.

>It's fundamentally impossible because when you publish the topics are going to be in a way or another niche enough that you know all of the people working on those topics Even if you know the people in the field and you can guess the authors, double blind does not hurt. Actually, I don’t see any downside of double blind. Actually, names are very often used to let you out of the field if you are not part of the usual…

You can't see any downside of double blind? There are many systems we use which provide anonymity, and they always result in some form of abuse. Such as phone numbers (scam calls), Internet (cyber harassment, scams, viruses), cryptocurrency (all the scams). Any system where humans can gain advantages from abusing anonymity they do so. I'm not an academic so maybe there are already safe guards, but from my understandi…

I don't understand what you're trying to insinuate and how negative effects can arise. We have double blind review in all major journals.

1. If a reviewer recognizes the author without declining the review, the result cannot be worse than without blind review. A difference would only be if the names of the reviewers are also not anonymous, but this creates much worse problems and would seriously jeopardize the review process.

2. If a reviewer doesn't recognize a dupe or paper very similar to one already published, then the reviewer is not competent and the journal has a problem with the reviewer pool or reviewer selection. That is a problem in any case, and is the reason why bad quality journals exist.

3. If the reviewer is incompetent, unfair or even insulting, then the author will complain to the editorial office who will then investigate and possibly get a third reviewer. The editor in chief or area editor can see the reviewer replies, just not the names of the reviewers and authors.

4. Your point about stealing research is not an issue. The publication is not anonymous; theft will be discovered very quickly by the scientific community (which will discover it more likely than two reviewers anyway). However, I'm pretty sure many top journals also use anti-plagiarism software.

I really don't see how double blind review could be abused more than non-blind review.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#239

Post by frankfrankfrank »

emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

Sorry, but there are several things problematic about that study, if your description is accurate.

Just one of the issues is the specific perception regarding that particular sport (basketball) and relative perceptions between sex differentiated similar sports (men’s and women’s basketball are not the same sport, albeit similar). A sport should have been chosen where there are as few differentiating perceptions as possible, which is nearly impossible, because men are inherently more competent at sport due to physiological realities. It bakes in a bias towards men, which is what you likely actually confirmed with the study you described.

Introducing sports alone essentially corrupted that research and likely biased the responses for relative innate competency of males in sports.

For example, ignoring its own challenges, what could have been done, is show video of a relatively broadly positively evaluated sport (not basketball), an activity that is also relatively broadly and positively evaluated female clustered competencies, e.g., effective child rearing, communicating effectively, conflict resolution, creative outputs, or even beauty pageants, etc., along with the inverse, i.e., poor performance of men in the same sport and poor performance of women in whatever is chosen, e.g., screaming and yelling at poorly behaved children.

It is in the past and by no means are or were you the only one who has long engaged in this type of poorly executed “research” that has merely been confirming researcher biases and often, thereby immensely damaging society, but maybe my illustration of a few of the several issues with what you described, will cause some changes in thinking.

There is a real hidden epidemic of not only single order thinking, but what really should be called negative order thinking in research, where research is not only not finding the truth, but rather even doing damage through confidence in false findings.

I say these things with an expensive relevant background, in the trenches of bad and … frankly … destructive research, if you will.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#240

Post by jonathanstrange »

pbhjpbhj wrote: post #237
jonathanstrange wrote: post #232

Earlier quoted context omitted.

Sure but the idea is that you don't try to find out who the author is, and if you recognize the author anyway, you decline the review. I review regularly and that's what I do whenever I recognize an author.

Don't you recognise authors on topics you're most competent to review on?

Not very often in a way that causes me to decline to review. What happens sometimes is that you have a suspicion but it's not definite. It could be one out of a dozen authors, or a newcomer you've never heard of. I'm in the humanities, however, things probably look different in the natural sciences.
Post reply on HN