Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Authors’ names have ‘astonishing’ influence on peer reviewers: study

nature.com

Post reply on HN
Print view View original

251–260 of 319 posts

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#251

Post by emacsen »

pbhjpbhj wrote: post #242
permo-w wrote: post #208

Earlier quoted context omitted.

the sports thing seems like a weird addition. did the text have some relevance to sport? if not it seems like the kind of thing that would skew results just by reminding the participants they’re in a weird study

Would be interesting to hear from OP, but I assumed it was a reminder to check ones stereotypes -- eg 'remember women can be good at traditionally male pursuits' (would work for some sports only).

It wasn't anything so complex as that.

The simpler version of the study had been done before, and the class I was a study design class. Our class assignment was to design, execute, and analyze a study.

We were simply combining two existing studies into a new study.

It bears mentioning- this was ~1996/1997 (I don't remember), I was a freshman, it was a single credit supplement, and we never published the results.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#252

Post by lupire »

caddemon wrote: post #245
jonathanstrange wrote: post #213

Earlier quoted context omitted.

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

Well in various CS fields people preprint on arXiv these days, so it is easy for reviewers to unblind themselves. Sometimes accidentally, because preprint timing and subsequent Twitter advertising coincide so well with conference deadlines...

Why doesn't arxiv allow name-blind preprint posting?

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#253

Post by lupire »

kalimanzaro wrote: post #236
epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I suspect the mechanism goes something like this: Reviewers prioritize their time above all, and after that correctness，leaving novelty in last place. If a paper comes from a unknown lab it tends to get greater scrutiny because a famous name is a subconscious stand-in for correctness. You spend less time reviewing famous authors because you think they are less likely to have bugs.

I think "aligned with reviewer's biases" is more important than correctness or novelty.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#254

Post by emacsen »

nicky0 wrote: post #203
emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

Can you elaborate on the sports thing? What effect did showing the sports had and do you have a thoory on the mechanism?

This is a really long time ago but if I remember, we ready a study about how sports effected perception. In the other study they did the male sport vs female sport thing, but not the name part, and so we just were combing two previous studies.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#255

Post by WithinReason »

erfgh wrote: post #248
andrepd wrote: post #228

Earlier quoted context omitted.

> To find any real bias you would have to assure the subjects that the researchers were equally accomplished. What do you mean? I think you didn't understand the experiment. Subjects were shown the same text with, at random, a male name, female name, or initials. These names were not real researchers, nor familiar to the subjects. > If it is known that most discoveries are made by males it is reasonable to give large…

You will find it beneficial to learn what a "prior probability distrubution" is.

It's actually you who doesn't understand probabilities. If a woman is twice as likely to make a discovery but women are only 5% of researchers, then it's both true that:

1. most discoveries are made by males

2. it is reasonable to give larger weight to new female-done research

contradicting you point.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#256

Post by lupire »

frankfrankfrank wrote: post #239
emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

Sorry, but there are several things problematic about that study, if your description is accurate. Just one of the issues is the specific perception regarding that particular sport (basketball) and relative perceptions between sex differentiated similar sports (men’s and women’s basketball are not the same sport, albeit similar). A sport should have been chosen where there are as few differentiating perceptions as po…

Why are you ignoring the stronger non-sports-relates results that PP reported? Your bias seems stronger than the bias you are complaining about.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#257

Post by emacsen »

goto11 wrote: post #168
DemeterFarm wrote: post #125

Earlier quoted context omitted.

Maybe someone in your field would know, is there any difference in the actual reliability/reproducibility difference in research done by men and women? Maybe you are simply measuring a real phenomenon but in a round about way?

The OP mention it is the same text which is rated differently depending on the gender of the author.

This was one of the few factors we kept the same- same text.

I think if it had been a larger study (not a one credit class run by 4 undergrads but an actual study with funding) we might have had multiple texts and randomized them, but we had one text- I don't even remember what the topic was.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#258

Post by lupire »

9wzYQbTYsAIc wrote: post #176
MilanTodorovic wrote: post #171

Earlier quoted context omitted.

The problem with that is, at least in my experience, that you read such things once from a reputable source and never check it again.

When I was studying undergrad psych, Psychology Today was falling into disrepute. Are they still a reputable source, today?

It can't be "still reputable" after "falling into disrepute".

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#259

Post by lupire »

erfgh wrote: post #248
andrepd wrote: post #228

Earlier quoted context omitted.

> To find any real bias you would have to assure the subjects that the researchers were equally accomplished. What do you mean? I think you didn't understand the experiment. Subjects were shown the same text with, at random, a male name, female name, or initials. These names were not real researchers, nor familiar to the subjects. > If it is known that most discoveries are made by males it is reasonable to give large…

You will find it beneficial to learn what a "prior probability distrubution" is.

You will find it beneficial to learn about "selection bias".

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#260

Post by emacsen »

concordDance wrote: post #184
emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

> Our small study showed what you'd expect- students- men and women students rated male writers more highly than female writers, and initials were right in the middle, though they tended to be more like responses for men. How much more? That's really important.

We didn't have a huge study, At the time, I felt like "Sexism is dead" and the university I was in was 2/3rds women to 1/3rd men, so we'd never find a bias.

As a young man (18/19) I learned to question my assumptions. I thought "It's the 90s, sexism is dead.", but it just wasn't true.

Post reply on HN