Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Authors’ names have ‘astonishing’ influence on peer reviewers: study

nature.com

Post reply on HN
Print view View original

261–270 of 319 posts

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#261

Post by seydor »

keithwinstein wrote: post #124

Double-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), systems (OSDI, SOSP, USENIX ATC, HotOS), security (Usenix Security, S&P), machine learning (NeurIPS and ICML), graphics and HCI (SIGGRAPH and CHI), and at least some top-tier theory conferences (like FOCS). So I think most computer scientists would agree with the…

When talking about peer review and open access, the reference point is life sciences, not computer science. CS is very different when it comes to the significance and process of those. In life sciences double blind is practically impossible because people know the people in their narrow field

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#262

Post by emacsen »

bitexploder wrote: post #112
emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

Another interesting bias along those lines: https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect I imagine these biases swing in all sorts of directions depending on the context. Some are intuitive, many are not.

I'm not a psychologist, I have nothing but that BA in the field, but if I had to make a guess, my guess is that "wonderful" is not the same as "competent".

Even at the time, there were studies that showed traits associated with women were rated more positively by men and women than traits associated with men.

The way you study this is you'd give a list of words:

gentle caring assertive stubborn aggressive loving

(ideally you'd randomize the word order too)

And then you'd have a group of subjects rate them as more associated with men or women on a scale.

Ideally you'd get a big sample and replicate this study.

Then either in the same study at a different time, or another study, you'd take those same words and you'd ask your subjects to rate them as positive or negative.

That's where you see effects like the one mentioned in the wikipedia article.

That kind of thing's been done a bunch of times.

But studies on gender and competency have been done too, and at the time they showed the same pattern as we found.

I say "At the time" because this is >20 years ago.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#263

Post by dmd »

rendall wrote: post #223
epolanski wrote: post #219

Earlier quoted context omitted.

> Don't you have blind peer review? It's fundamentally impossible because when you publish the topics are going to be in a way or another niche enough that you know all of the people working on those topics and conferences make those circles even more public. Say that you are studying e.g. "helical peptides", which is a vague and broad term on one side, on the other one, all the people that work on specific aspects o…

But this is contradictory. Can you explain? On the one hand are famous scientists who have wide influence, on the other hand are small academic circles working on obscure, niche topics where everyone knows each other so well that blind review is impractical. How can one scientist be both so reknowned that their name tilts the balance of grant money unfairly away from other less known researchers, and at the same time…

I can name a dozen people who, for anyone in my field, are extremely famous - to the point of if you meet and talk to them at a conference, it'll be the talk of the lab for weeks afterwards and everyone will gush about how cool it is that you got to meet them and talk to them.

If I walk two corridors over - still in my department, but just a slightly different aspect of the field - they won't even recognize the name.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#264

Post by caddemon »

lupire wrote: post #252
caddemon wrote: post #245

Earlier quoted context omitted.

Well in various CS fields people preprint on arXiv these days, so it is easy for reviewers to unblind themselves. Sometimes accidentally, because preprint timing and subsequent Twitter advertising coincide so well with conference deadlines...

Why doesn't arxiv allow name-blind preprint posting?

That would be a good idea, and then have the names revealed in a future version like the way the paper itself can be updated. I don't think it is possible though. If it is, it certainly isn't used.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#265

Post by hnaccount_rng »

mikk14 wrote: post #204
hnaccount_rng wrote: post #177

Earlier quoted context omitted.

But as a (final) reader I would weigh supporting evidence by the same author/author group/lab far weaker than supporting evidence by unrelated sources! That being said: One could get around that by replacing the citation style for the review version only. There are typically changes anyways and the given amount of fights that I had with editors for pure typesetting or "improvements of english" that one would actually…

As the final reader you're expected to do some due diligence and at the bare minimum read the reference list. Once you do, you can easily see which papers are self citations and which ones are not. If you're not doing this due diligence you probably don't care too much about this paper anyway and there is no harm done if you think a reference is not a self citation when it actually is.

That doesn't help if references are given as First Author et al. or even as [number] and while I agree in principle it simply doesn't happen in practice.

But then it just weakens your statement to make it worse. And I'm not even sure if it makes a difference for what the article talks about. Because the reviewer needs to actively go looking. And at least to my understanding these effects are not due to people going out of their way to misjudge people, but rather that it's an effect of subconscious prejudices. And for the latter breaking the obvious connection is probably enough.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#266

Post by rusticpenn »

jonathanstrange wrote: post #232
rusticpenn wrote: post #221

Earlier quoted context omitted.

You can easily find the authors based on the references they use and the topic.

Sure but the idea is that you don't try to find out who the author is, and if you recognize the author anyway, you decline the review. I review regularly and that's what I do whenever I recognize an author.

Not everyone seems to have your ethics.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#267

Post by avidphantasm »

jonathanstrange wrote: post #213
epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

Even with blind reviews, many people cite their prior work extensively, so it can be hard to truly obscure who the paper authors are.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#268

Post by ljw1001 »

caddemon wrote: post #264
lupire wrote: post #252

Earlier quoted context omitted.

Why doesn't arxiv allow name-blind preprint posting?

That would be a good idea, and then have the names revealed in a future version like the way the paper itself can be updated. I don't think it is possible though. If it is, it certainly isn't used.

Without peer review, publications without names would be untrusted and untrustworthy, and would probably represent a security issue

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#269

Post by jcfrei »

jonathanstrange wrote: post #213
epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

Heard from colleagues working in academia that their profs were absolutely strict about the layout, the way graphs look, the colors which needed to be used, etc. All so reviewers could decode who's actually behind a paper.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#270

Post by SkyBelow »

Orothrim wrote: post #229
techas wrote: post #220

Earlier quoted context omitted.

>It's fundamentally impossible because when you publish the topics are going to be in a way or another niche enough that you know all of the people working on those topics Even if you know the people in the field and you can guess the authors, double blind does not hurt. Actually, I don’t see any downside of double blind. Actually, names are very often used to let you out of the field if you are not part of the usual…

You can't see any downside of double blind? There are many systems we use which provide anonymity, and they always result in some form of abuse. Such as phone numbers (scam calls), Internet (cyber harassment, scams, viruses), cryptocurrency (all the scams). Any system where humans can gain advantages from abusing anonymity they do so. I'm not an academic so maybe there are already safe guards, but from my understandi…

>Truly double blind peer review would likely encourage this. Maybe it would also further encourage people to steal research or peer review outside of their expertise.

How so?

For smaller papers, if the papers still stand up to review even when the research has been cut up, that seems a neutral effect at worse. It might even be seen as a benefit as each paper is more focused.

As for stealing results, the names will still be attached before the printing is done. Only the review process will be double blind. Thus any ability to detect stolen research will still occur.

Post reply on HN