Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Authors’ names have ‘astonishing’ influence on peer reviewers: study

nature.com

Post reply on HN
Print view View original

211–220 of 319 posts

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#211

Post by throw827474737 »

SapporoChris wrote: post #206
bitexploder wrote: post #112

Earlier quoted context omitted.

Another interesting bias along those lines: https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect I imagine these biases swing in all sorts of directions depending on the context. Some are intuitive, many are not.

How do you reconcile the https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect with https://en.wikipedia.org/wiki/Gender_pay_gap

Not needed because "women are wonderful" is to some degree mostly orthogonal to "men are more professional/deserve more/"... wonderful good attributes are not needed or even to some degree counter to "high performance" (but depends a lot where we look at, sure!)

Also pay-gap has likely also to do with the current men-bias also? Wonder if that would be the same if we had 90% woman throughout in these decising positions..

In the end all mine assumptions, what I actually want to say is: I don't think those two contradict or relate at all.. what GP already said well with "biases swing in all sorts of directions depending on the context"

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#212

Post by ambicapter »

SapporoChris wrote: post #206
bitexploder wrote: post #112

Earlier quoted context omitted.

Another interesting bias along those lines: https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect I imagine these biases swing in all sorts of directions depending on the context. Some are intuitive, many are not.

How do you reconcile the https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect with https://en.wikipedia.org/wiki/Gender_pay_gap

Not GP, but:

From your wikipedia link

> There are two distinct numbers regarding the pay gap: non-adjusted versus adjusted pay gap. The latter typically takes into account differences in hours worked, occupations chosen, education and job experience.[1] In the United States, for example, the non-adjusted average woman's annual salary is 79% of the average man's salary, compared to 95% for the adjusted average salary.[2][3][4][5]

The remaining 5% could be from the "women are wonderful" positive attributes not being the narrow selection of ones that are highly sought after in well-renumerated jobs, or (spitballing here) psychosocial (Expectations of a pay gap driving negotiation behavior or something).

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#213

Post by jonathanstrange »

epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I don't understand this. Don't you have blind peer review?

In my discipline, most journals even have double blind peer review. I thought that's the standard.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#214

Post by erfgh »

emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

If it is known that most discoveries are made by males it is reasonable to give larger weight to new male-done research, if the only thing that is known about the researchers is their gender and if the reader is not enough of an expert on the subject matter to treat it 100% on its own merit.

I don't see this "bias" as inefficient or counter-productive. It is just an artifact of the way you designed your flawed experiment.

To find any real bias you would have to assure the subjects that the researchers were equally accomplished.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#215

Post by triknomeister »

jonathanstrange wrote: post #213
epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

Mathematics doesn't have double blind peer review unfortunately. It will help in a big number of cases, even if some cases fall off (for example, if you do experiments on a well-known software which only one lab works on, then double blind will have a limited impact.)

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#216

Post by tovej »

erfgh wrote: post #214
emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

If it is known that most discoveries are made by males it is reasonable to give larger weight to new male-done research, if the only thing that is known about the researchers is their gender and if the reader is not enough of an expert on the subject matter to treat it 100% on its own merit. I don't see this "bias" as inefficient or counter-productive. It is just an artifact of the way you designed your flawed experi…

An experiment like OP's would 100% show bias. There weren't multiple researchers, each reader read the exact same paper, but with a different gendered (or non-gendered) name as the author.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#217

Post by triknomeister »

YakBizzarro wrote: post #195
keithwinstein wrote: post #124

Double-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), systems (OSDI, SOSP, USENIX ATC, HotOS), security (Usenix Security, S&P), machine learning (NeurIPS and ICML), graphics and HCI (SIGGRAPH and CHI), and at least some top-tier theory conferences (like FOCS). So I think most computer scientists would agree with the…

In my experience, at least in the field of Solid-state Physics, double-blind review is quite rare. Also, it would be challenging to implement, since it's often very clear who is the group that submitted a paper. The community is not so big, everyone know everyone, samples and techniques are quite peculiar for each group and a paper typically cite a lot of previous work, to avoid large repetitions. It's not uncommon t…

The point is that, there are no downsides to double-blind peer review, and it is hence possible that with double blind peer review, less well-known people might get more journal articles. This might lead to more diverse research techniques being used since you don't need a big name to publish anymore. And then, in a few years, you might find it hard to guess the groups from techniques, because a single group might have multiple different techniques.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#218

Post by KvanteKat »

jonathanstrange wrote: post #213
epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

As the linked article points out, practices can vary widely across fields (and in fields where preprints are available, maintaining authour anonymity in a double-blind setting furthermore relies on revieweres not having come across a preprint of the paper that is being reviewed which in turn can easily happen in smaller research-communities).

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#219

Post by epolanski »

jonathanstrange wrote: post #213
epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

> Don't you have blind peer review?

It's fundamentally impossible because when you publish the topics are going to be in a way or another niche enough that you know all of the people working on those topics and conferences make those circles even more public.

Say that you are studying e.g. "helical peptides", which is a vague and broad term on one side, on the other one, all the people that work on specific aspects of helical peptides (say, using them to bind surfaces that are very different in polarization, e.g. batteries) know each other and blind review would add nothing really. Even when it is blind you likely know who's reviewing it most of the time.

There are topics that are of course wider.

To make a comparison with programming: imagine you're involved in an open source community with a language that is niche enough that you know most of the people involved in it (it is common for many lisp dialects).

You can likely be given a program in that language and figure out likely who wrote it due to their interests, programming or api designing, or the use of static types, or some kind of linting that you know only a handful of people use, background, etc.

With the huge difference that in case of science the circles are much smaller because you know all your peers are in academia, who of them has specific instruments (not everyone has the same lab equipment), but who doesn't also have some others, you may know that some of your peers have their niche and "main branches" of research so you figure those things much easier. Also, in those circles you also know who is involved in your "code reviews" so if you know that A and B aren't reviewing it (you know they would've told you) it can only be C, D and E, but the feedback requesting this kind of experiment could only come from "C"...

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#220

Post by techas »

epolanski wrote: post #219
jonathanstrange wrote: post #213

Earlier quoted context omitted.

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

> Don't you have blind peer review? It's fundamentally impossible because when you publish the topics are going to be in a way or another niche enough that you know all of the people working on those topics and conferences make those circles even more public. Say that you are studying e.g. "helical peptides", which is a vague and broad term on one side, on the other one, all the people that work on specific aspects o…

>It's fundamentally impossible because when you publish the topics are going to be in a way or another niche enough that you know all of the people working on those topics

Even if you know the people in the field and you can guess the authors, double blind does not hurt. Actually, I don’t see any downside of double blind.

Actually, names are very often used to let you out of the field if you are not part of the usual family…

Post reply on HN