Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Authors’ names have ‘astonishing’ influence on peer reviewers: study

nature.com

Post reply on HN
Print view View original

171–180 of 319 posts

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#171

Post by MilanTodorovic »

jdkoeck wrote: post #166
9wzYQbTYsAIc wrote: post #140

Earlier quoted context omitted.

That reminds me of a bias study where the researchers were investigating the effects of stereotypes. They told one group of Asian women that “women are worse at math than men” and another group of Asian women that “Asians are better at math than non-Asians”. Both are common stereotypes. They then measured how well the two groups did on subsequent math exercises. Interestingly, they found that the latter group (positi…

That’s called “stereotype threat”, and it’s been caught up in the replication crisis. In short, the effect is hard to reproduce and tends to be small. It’s been known for some time now, here’s an article from 2015: https://www.psychologytoday.com/us/blog/rabble-rouser/201512... I wish people would stop bringing up studies that don’t reproduce, they’re no better than anecdotes.

The problem with that is, at least in my experience, that you read such things once from a reputable source and never check it again.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#172

Post by keithwinstein »

Ma8ee wrote: post #141
keithwinstein wrote: post #124

Double-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), systems (OSDI, SOSP, USENIX ATC, HotOS), security (Usenix Security, S&P), machine learning (NeurIPS and ICML), graphics and HCI (SIGGRAPH and CHI), and at least some top-tier theory conferences (like FOCS). So I think most computer scientists would agree with the…

I’m confused about your second point: isn’t the reader the one who gains the most from reading the best papers, independently of who’s the author?

For the most part, sure, but at the margin I'm not sure the "best paper" can be completely well-defined within its four corners, independent of the author. E.g. the reader might rationally care (to some degree) about what a paper says about the future of the field. If an underfunded non-famous researcher describes marginal preliminary results by pursing some approach, that's one thing, but if my very successful colleagues at Berkeley/Washington/Tsinghua or Google/MSR/Alibaba publish a paper that's like, "We have gotten pretty excited about this technique for a bunch of reasons, some quantified but some more conjectural, or qualitative, or just more difficult to measure so not done yet, but here are some interesting preliminary measurements that should whet your appetite, and because of who we are, our track record of success, and our many grad students/researchers digging for gold, we'll probably find more to report in due time" well, am I wrong to be a little more interested in the second paper on the grounds that I'd like to keep up with the field?

(You might also reasonably believe that the job of an editor or program committee is to assemble the most edifying program or issue when considered as an ensemble, rather than each paper individually.)

I also don't think it's completely crazy to consider to a tiny degree what the reader "wants," separate from their true interest in an omniscient sense. When you click around on the Internet, do you only read the "best papers"? Probably not. This is similar to why the New York Times (and Hacker News) don't use 90% of their word count to remind readers to eat healthy, stop smoking, maintain friendships and regular activity, sleep regular hours, and give away most of their money to buy malaria nets.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#174

Post by 9wzYQbTYsAIc »

inglor_cz wrote: post #165
9wzYQbTYsAIc wrote: post #140

Earlier quoted context omitted.

That reminds me of a bias study where the researchers were investigating the effects of stereotypes. They told one group of Asian women that “women are worse at math than men” and another group of Asian women that “Asians are better at math than non-Asians”. Both are common stereotypes. They then measured how well the two groups did on subsequent math exercises. Interestingly, they found that the latter group (positi…

The experiment you discuss is called "stereotype threat" and it belongs into the huge set of psychological results that cannot be replicated. [1] It seems that there is a bias against publishing studies with null results, which skews the overall picture both among the researchers and in the media. [1] https://www.tandfonline.com/doi/full/10.1080/23743603.2018.1...

Interesting, good to know, thanks.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#175

Post by Nokinside »

themitigating wrote: post #88
Nokinside wrote: post #71

Every paper Albert Einstein submitted to anonymous peer review was rejected. (he had only one peer anonymous reviewed paper and it had an error)

I'm confused because of your second sentence but according to this site https://mindmatters.ai/2020/05/einsteins-only-rejected-paper... ". Albert Einstein only had one anonymous peer review in his career — and the paper was rejected2. This happened in 1936." How is just one paper submitted anonymously being rejected an indication of a trend?

There was no trend.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#176

Post by 9wzYQbTYsAIc »

MilanTodorovic wrote: post #171
jdkoeck wrote: post #166

Earlier quoted context omitted.

That’s called “stereotype threat”, and it’s been caught up in the replication crisis. In short, the effect is hard to reproduce and tends to be small. It’s been known for some time now, here’s an article from 2015: https://www.psychologytoday.com/us/blog/rabble-rouser/201512... I wish people would stop bringing up studies that don’t reproduce, they’re no better than anecdotes.

The problem with that is, at least in my experience, that you read such things once from a reputable source and never check it again.

When I was studying undergrad psych, Psychology Today was falling into disrepute.

Are they still a reputable source, today?

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#177

Post by hnaccount_rng »

jedberg wrote: post #143
seanmcdirmid wrote: post #139

Earlier quoted context omitted.

Peer reviews are usually blind (authors don’t know who their reviewers are). What you are referring to is double blind (reviewers don’t know who the authors are). The challenge with double blind is anonymizing content that often relies on cited prior work.

> The challenge with double blind is anonymizing content that often relies on cited prior work. You're the second person to bring this up, but I'm not sure why it's a problem. Just don't say "based on my own previous work" and instead say "based on previous work". I.e. cite yourself the same way that you'd cite anyone else's work.

But as a (final) reader I would weigh supporting evidence by the same author/author group/lab far weaker than supporting evidence by unrelated sources! That being said: One could get around that by replacing the citation style for the review version only. There are typically changes anyways and the given amount of fights that I had with editors for pure typesetting or "improvements of english" that one would actually be helpful to do.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#178

Post by thomasahle »

Ma8ee wrote: post #141
keithwinstein wrote: post #124

Double-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), systems (OSDI, SOSP, USENIX ATC, HotOS), security (Usenix Security, S&P), machine learning (NeurIPS and ICML), graphics and HCI (SIGGRAPH and CHI), and at least some top-tier theory conferences (like FOCS). So I think most computer scientists would agree with the…

I’m confused about your second point: isn’t the reader the one who gains the most from reading the best papers, independently of who’s the author?

It's more like how some people would prefer to watch the next episode of the show they are already watching, even if some episode of another show is objectively better.

Or some would rather read a mediocre Tweet from somebody they follow than a good tweet from somebody they don't know.

That's why OP calls it "eating your vegetables", forcing people to read the "best" articles rather than the articles they "want" to read.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#179

Post by eldenwrong »

nextlevelwizard wrote: post #155
jedberg wrote: post #121

It blows my mind that peer reviews aren't done blind. Not only is there this geneder bias but also bias of "I know that person they're famous I'm sure their work is good" or "I've rejected their paper in the past". Seems like an easy fix would be to make all peer reviews blind. In fact, for any study that gets federal funding, they should have to publish their hypothesis ahead of time into an escrow system, submit th…

Academics just like every other aspect of human life is full of politics and this includes peer reviews as well. Papers regularly get panned for variety of reasons that have nothing to do with actual content of the paper and maybe even worse sometimes because the reviewer happen to be working on something similar and want secure a bigger grant for themselves.

That's false. Trust the science.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#180

Post by LtWorf »

magicalhippo wrote: post #78

A while back I subscribed for The Economist on a whim. I'm not really into economics, but I felt they provided a nice alternative view of what was going on elsewhere in the world compared to what the local news here in Norway reported. One thing I quickly noticed was that the articles were never signed by the authors directly. I found this striking, but also refreshing. It caused me to pay more attention to what was…

On the other hand, if an author is consistently bad, signing would allow you to skip the content.
Post reply on HN