Authors’ names have ‘astonishing’ influence on peer reviewers: study • Hacker News Board

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#271

Post by bazoom42 » Wed, Oct 12, 2022, 12:56 PM UTC

erfgh wrote: post #214

emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

If it is known that most discoveries are made by males it is reasonable to give larger weight to new male-done research, if the only thing that is known about the researchers is their gender and if the reader is not enough of an expert on the subject matter to treat it 100% on its own merit. I don't see this "bias" as inefficient or counter-productive. It is just an artifact of the way you designed your flawed experi…

> If it is known that most discoveries are made by males it is reasonable to give larger weight to new male-done research

No it isn’t. Dont you see this is just circular reasoning?

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#272

Post by SkyBelow » Wed, Oct 12, 2022, 12:59 PM UTC

emacsen wrote: post #262

bitexploder wrote: post #112

Earlier quoted context omitted.

Another interesting bias along those lines: https://en.m.wikipedia.org/wiki/Women-are-wonderful_effect I imagine these biases swing in all sorts of directions depending on the context. Some are intuitive, many are not.

I'm not a psychologist, I have nothing but that BA in the field, but if I had to make a guess, my guess is that "wonderful" is not the same as "competent". Even at the time, there were studies that showed traits associated with women were rated more positively by men and women than traits associated with men. The way you study this is you'd give a list of words: gentle caring assertive stubborn aggressive loving (ide…

>my guess is that "wonderful" is not the same as "competent".

Right. There are two different sets of benefits that help or hurt someone in different ways. Being competent when standing trial can work against you while being wonderful will reduce the risk of a conviction and reduce the sentence if you are convicted.

We have identified the "competent" bias and are taking steps to correct it, but we need to do the same with the "wonderful" bias in other systems. For starters we need to recognize how strong that bias is in certain fields. For one example, there are specific crimes that people would bet are extremely gendered in nature, and the crime statistics show they would be right, but interviewing the population at large and querying victims, including those who never went to the police or who were turned away by the police (or even worse, who couldn't legally be victims because of how biased even the laws are), we see the gender component goes away. The rate of men victimized by women and women victimized by men are at near a 50/50 ration (I think 49.8 to 50.2).

Even the extent of studies measuring the impact of the wonderful effect is lacking compared to studies measuring the competent effect (which itself is likely a bias of the wonderful effect).

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#273

Post by Stupulous » Wed, Oct 12, 2022, 1:05 PM UTC

erfgh wrote: post #214

emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

If it is known that most discoveries are made by males it is reasonable to give larger weight to new male-done research, if the only thing that is known about the researchers is their gender and if the reader is not enough of an expert on the subject matter to treat it 100% on its own merit. I don't see this "bias" as inefficient or counter-productive. It is just an artifact of the way you designed your flawed experi…

Even if one sex was 10x as likely as the other to produce good research, it is still sexist bias to judge research on the basis of the author's sex. Fairness and meritocracy mean everyone gets a chance; no one should be stopped from succeeding because they belong to a low-performing group.

I think it's safe to say that this study can be treated as an anecdote because we don't know the effect size or exact methodology and it was never peer-reviewed. And I'd agree with your point if we were talking about the Nobel prize study, which evaluates people as individuals. But arguing that 'real bias' is bias that doesn't come from empirical evidence doesn't make a whole lot of sense- all bias comes from empirical evidence of varying quality.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#274

Post by fsociety999 » Wed, Oct 12, 2022, 1:25 PM UTC

themitigating wrote: post #115

Test0129 wrote: post #104

Earlier quoted context omitted.

HN isn't a journal and so academic sources of evidence really are overkill. However, I'll throw you a bone. Go to your favorite major journal and start looking at the "conflicts of interest" and "grants" section. Once you take money from someone (except for NIST, in my experience) you're basically beholden to try your hardest to get the results they're looking for. Some scientists are moral enough to still return bad…

"HN isn't a journal and so academic sources of evidence really are overkill" I disagee, you should backup your accusations or statements (unless widly accepted) with some reputable source. This person claimed that science was bought and paid for. Did they mean all of it? Or most? That's an insane accusation that requires evidence. "Go to your favorite major journal and start looking at the "conflicts of interest" and…

Is it really that insane of an accusation? This is how almost everything in the world works.

This isn’t only about greed either. People want their research published for reasons other than greed. For example, they want to move up in their career or achieve recognition.

After looking at a lot of medical studies related to COVID during the last couple years, I have seen first hand how biased and inaccurate many of them are. Some of these studies are even mentioned in major news outlet despite their obvious flaws when you actually begin to scrutinize them. Think big pharma providing research grants for studies that conclude their products are effective.

The OP never said that people falsify data as a result of receiving grants from interested parties. They often don’t have to. They can simply design the experiment in a way that doesn’t account for specific variables or behaviors then use the resulting data to reach a specific conclusion.

I remember seeing an article related to AI research on HN a little while ago that somewhat explained this problem. The grant money all goes to people researching deep neural networks which creates a reinforcing feedback loop. Since all the money goes to one branch of research, it creates very few opportunities to research competing ideas. I believe it was this one:

https://nautil.us/deep-learning-is-hitting-a-wall-14467/

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#275

Post by cozzyd » Wed, Oct 12, 2022, 1:36 PM UTC

hedora wrote: post #108

cozzyd wrote: post #47

Earlier quoted context omitted.

It's just that experiments in astroparticle physics are expensive and have lots of people on them. Like, how is the IceCube Collaboration supposed to write an anonymous paper? Even the most cursory description of the detector would give it away...

Well, maybe there will be many IceCube's later? (Probably not, but I'm routinely shocked when I learn about new giant testbeds in my field...) Alternatively, when IceCube 2 comes out, the old IceCube crowd might be focusing on other stuff, and not paying attention to the IceCube 2 politics. That makes them great peer reviewers (no horse in the race, but knowledgeable).

The proposed IceCube Gen2 is mostly a superset and only slightly disjoint with IceCube (for example, I am not an IceCube collaborator but I am on Gen2...). But the point is that for experiments that are larger than a few PI's, anonymity of authors is basically impossible (since all papers have everyone on them).

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#276

Post by dataflow » Wed, Oct 12, 2022, 1:53 PM UTC

jonathanstrange wrote: post #213

epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I don't understand this. Don't you have blind peer review? In my discipline, most journals even have double blind peer review. I thought that's the standard.

Not every conference or journal is blinded, no.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#277

Post by insane_dreamer » Wed, Oct 12, 2022, 1:57 PM UTC

Blikkentrekker wrote: post #153

insane_dreamer wrote: post #129

Earlier quoted context omitted.

HN doesn't allow you to follow users--that makes a big difference in putting the focus more on the content than the users.

Peer-reviewers also don't follow particular names around; they see the name above the submission and it influences them, as on H.N. no doubt. But it's far worse, even without a recognized names, most votes are cast without proper reading, and I'm fairly certain also by the least intelligent subsection given how often submissions are upvoted on H.N. and Reddit that are pure clickbait and demolished in the comments by…

> Peer-reviewers also don't follow particular names around; they see the name above the submission and it influences them, as on H.N. no doubt.

Agreed. I was referring to Quora.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#278

Post by Al-Khwarizmi » Wed, Oct 12, 2022, 1:57 PM UTC

keithwinstein wrote: post #124

Double-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), systems (OSDI, SOSP, USENIX ATC, HotOS), security (Usenix Security, S&P), machine learning (NeurIPS and ICML), graphics and HCI (SIGGRAPH and CHI), and at least some top-tier theory conferences (like FOCS). So I think most computer scientists would agree with the…

> authors are not the only beneficiaries of a scientific publication. The interests of the reader matter too -- the journal or conference has some duty to serve them.

> I don't think we should act like scientific publication is only to give authors a line on their CV and the readers' preference is 100% irrelevant

In many fields, the reader can go and read what they want in arXiv or a similar repository. And in the fields where this is not the case, it should be. The elite authors that you mention, in particular, shouldn't have any problem to get their papers read by linking them in social networks, starting a blog, etc.

While this wasn't the case 50 years ago, right now almost no one reads journals from front to back, people just search for individual papers, and the main purpose of the peer-review process of conferences and journals is basically gatekeeping and providing some signal for career evaluation, i.e., to "give authors a line on their CV". Thus, I don't think there is any reason to judge anything but the paper contents.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#279

Post by cycomanic » Wed, Oct 12, 2022, 1:58 PM UTC

lou1306 wrote: post #197

cycomanic wrote: post #159

Earlier quoted context omitted.

I disagree, the style to write as a "neutral" observer (often writing in passive voice) is frowned upon nowadays for good reason. Part of that is also that the information if the authors wrote a citation or not can be important to a reviewer. For example it is unfortunately quite common that authors publish results in a salami tactic to maximize the number of publications. There can be a significant difference in imp…

> For example it is unfortunately quite common that authors publish results in a salami tactic to maximize the number of publications Why is this unfortunate? I'd argue that splitting results in multiple publications is a) Riskier for authors (higher chance of rejection) and b) More convenient for readers (each paper requires less mental load, being focused on a single aspect). So, even if there's a payoff for author…

The tactic is more advantageous to authors, because they get more articles (which is used as a metric to evaluate scientific success) and I would argue it's less risky. Say you split up your results up into 3 papers, your chance of one paper being rejected might be higher, however your chance of all papers being rejected is lower.

In terms of more convenient for readers, you discount the mental load required in finding papers. That's in fact one of the biggest problems in many scientific fields at the moment. There are so many papers being published that it is very hard to keep up with the field. Reading the same amount of results also requires a much higher load, because if authors split up the results into 3 papers, the individual papers are not suddenly shorter, but in fact the overall page count is typically almost 3 times of a paper that would have put everything into a single paper.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#280

Post by mattkrause » Wed, Oct 12, 2022, 1:59 PM UTC

kalimanzaro wrote: post #236

epolanski wrote: post #209

As a former researcher with multiple high-impact publications I can 100% confirm it: - when I was a researcher in Italy for an unknown lab, all of our articles were generally very scrutinized and went through years of reviews before publication - when I worked in Michal Graetzel[1]'s laboratory things get published much more easily on more higher impact journals with less scrutiny. Not only most of the publications d…

I suspect the mechanism goes something like this: Reviewers prioritize their time above all, and after that correctness，leaving novelty in last place. If a paper comes from a unknown lab it tends to get greater scrutiny because a famous name is a subconscious stand-in for correctness. You spend less time reviewing famous authors because you think they are less likely to have bugs.

Reviewing a paper is effectively unrewarded: you need some kind of reviewing service on your CV, but the amount and quality is barely measured, let alone considered for career progression.

Reviewing a very good paper is quick and easy ("LGTM"). Really terrible papers are not too bad if they're obviously awful, but reviewing something subtly flawed takes a lot of work: you need to identify the flaws and describe them in a way that's compelling enough to convince the authors--or at least the editors. Ideally, you'll also explain how to address them, which is more work now and down the road when you review the authors' often-grudging implementation of your suggestion.

In such a world, people may be increasingly reluctant to review papers without some indication of their quality (for which name is a rough proxy). My solution to this is to somehow make reviewing more valued. It's an important part of science and deserves more than a checkbox.