Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Authors’ names have ‘astonishing’ influence on peer reviewers: study

nature.com

Post reply on HN
Print view View original

241–250 of 319 posts

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#241

Post by pbhjpbhj »

concordDance wrote: post #184
emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

> Our small study showed what you'd expect- students- men and women students rated male writers more highly than female writers, and initials were right in the middle, though they tended to be more like responses for men. How much more? That's really important.

Yes, but also is that "what you'd expect", why?

Most things I read I have no idea of the sex of the author. Do people even look at author names before they start reading (online, say), and even then they can be nom de plumes, or non-gendered names (sometimes surprisingly).

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#242

Post by pbhjpbhj »

permo-w wrote: post #208
emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

the sports thing seems like a weird addition. did the text have some relevance to sport? if not it seems like the kind of thing that would skew results just by reminding the participants they’re in a weird study

Would be interesting to hear from OP, but I assumed it was a reminder to check ones stereotypes -- eg 'remember women can be good at traditionally male pursuits' (would work for some sports only).

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#243

Post by joenot443 »

SapporoChris wrote: post #206
bitexploder wrote: post #112

Earlier quoted context omitted.

Another interesting bias along those lines: https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect I imagine these biases swing in all sorts of directions depending on the context. Some are intuitive, many are not.

How do you reconcile the https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect with https://en.wikipedia.org/wiki/Gender_pay_gap

They're two related but ultimately very different phenomena, they don't necessarily need reconciliation.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#244

Post by keithwinstein »

chalst wrote: post #222
keithwinstein wrote: post #124

Double-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), systems (OSDI, SOSP, USENIX ATC, HotOS), security (Usenix Security, S&P), machine learning (NeurIPS and ICML), graphics and HCI (SIGGRAPH and CHI), and at least some top-tier theory conferences (like FOCS). So I think most computer scientists would agree with the…

Double-blinding is not common in PL theory and implementation.

I can't speak to every PL venue, but all four places that csrankings.org lists for PL are double-blind these days (PLDI, POPL, ICFP, and OOPSLA).

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#245

Post by caddemon »

jonathanstrange wrote: post #213
epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

Well in various CS fields people preprint on arXiv these days, so it is easy for reviewers to unblind themselves. Sometimes accidentally, because preprint timing and subsequent Twitter advertising coincide so well with conference deadlines...

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#247

Post by concinds »

Ma8ee wrote: post #141
keithwinstein wrote: post #124

Double-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), systems (OSDI, SOSP, USENIX ATC, HotOS), security (Usenix Security, S&P), machine learning (NeurIPS and ICML), graphics and HCI (SIGGRAPH and CHI), and at least some top-tier theory conferences (like FOCS). So I think most computer scientists would agree with the…

I’m confused about your second point: isn’t the reader the one who gains the most from reading the best papers, independently of who’s the author?

Established experts have a certain "weight"; if they decide to endorse a fringe or cutting-edge scientific view, that would be in many people's interest to know.

But if that same paper in support of a fringe theory is reviewed under an anonymous name, it might not get published, even if it's perfectly well-argued, simply because "the topic is fringe and not very relevant to our readers" (circular reasoning).

And even if it's not fringe, how we view an author may influence whether their paper passes peer review for perfectly legitimate reasons. If a random scientist is the 100th one to weigh in on any old debate (let's say Penrose-Lucas), it may get dismissed as non-notable. But Penrose giving his updated thoughts would be notable.

There's costs to anonymous peer reviews. Credibility and notability can't be fully detached from the author.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#248

Post by erfgh »

andrepd wrote: post #228
erfgh wrote: post #214

Earlier quoted context omitted.

If it is known that most discoveries are made by males it is reasonable to give larger weight to new male-done research, if the only thing that is known about the researchers is their gender and if the reader is not enough of an expert on the subject matter to treat it 100% on its own merit. I don't see this "bias" as inefficient or counter-productive. It is just an artifact of the way you designed your flawed experi…

> To find any real bias you would have to assure the subjects that the researchers were equally accomplished. What do you mean? I think you didn't understand the experiment. Subjects were shown the same text with, at random, a male name, female name, or initials. These names were not real researchers, nor familiar to the subjects. > If it is known that most discoveries are made by males it is reasonable to give large…

You will find it beneficial to learn what a "prior probability distrubution" is.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#249

Post by emacsen »

pbhjpbhj wrote: post #241
concordDance wrote: post #184

Earlier quoted context omitted.

> Our small study showed what you'd expect- students- men and women students rated male writers more highly than female writers, and initials were right in the middle, though they tended to be more like responses for men. How much more? That's really important.

Yes, but also is that "what you'd expect", why? Most things I read I have no idea of the sex of the author. Do people even look at author names before they start reading (online, say), and even then they can be nom de plumes , or non-gendered names (sometimes surprisingly).

We announced the name of the author. The script went something like "We're going to ask you to read this piece, written by J. Smith.".

During debriefing, we would tell the subjects what we were actually looking for (perceptions based on gender) and most reported not remembering/caring about the gender of the author, but nonetheless the results were clear that there was a gender bias, regardless of whether they reported remembering or caring about the gender of the author.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#250

Post by lupire »

Orothrim wrote: post #229
techas wrote: post #220

Earlier quoted context omitted.

>It's fundamentally impossible because when you publish the topics are going to be in a way or another niche enough that you know all of the people working on those topics Even if you know the people in the field and you can guess the authors, double blind does not hurt. Actually, I don’t see any downside of double blind. Actually, names are very often used to let you out of the field if you are not part of the usual…

You can't see any downside of double blind? There are many systems we use which provide anonymity, and they always result in some form of abuse. Such as phone numbers (scam calls), Internet (cyber harassment, scams, viruses), cryptocurrency (all the scams). Any system where humans can gain advantages from abusing anonymity they do so. I'm not an academic so maybe there are already safe guards, but from my understandi…

If you are worried about "stealing research" , you have no right to public grant funding.
Post reply on HN