Double-anonymous/blind reviewing is completely standard in most areas of computer science, e.g. networking (SIGCOMM, NSDI, IMC, HotNets, MobiCom, MobiSys), systems (OSDI, SOSP, USENIX ATC, HotOS), security (Usenix Security, S&P), machine learning (NeurIPS and ICML), graphics and HCI (SIGGRAPH and CHI), and at least some top-tier theory conferences (like FOCS). So I think most computer scientists would agree with the…
Authors’ names have ‘astonishing’ influence on peer reviewers: study
141–150 of 319 posts
Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study
#142Very common for anyone who has ever spent time in academia. When I was still in grad school I noticed a trend in some major mathematical and CS journals where a lot of time it would be the same N people writing articles. This was driven home when my advisor told me "without such and such's name this paper has no chance of making it past review". Non-sense. However, for all the people that say "trust the science" this…
First of all, Knuth would never publish something "garbage" because his reputation would be on the line for everyone to read (forever).
Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study
#143It blows my mind that peer reviews aren't done blind. Not only is there this geneder bias but also bias of "I know that person they're famous I'm sure their work is good" or "I've rejected their paper in the past". Seems like an easy fix would be to make all peer reviews blind. In fact, for any study that gets federal funding, they should have to publish their hypothesis ahead of time into an escrow system, submit th…
Peer reviews are usually blind (authors don’t know who their reviewers are). What you are referring to is double blind (reviewers don’t know who the authors are). The challenge with double blind is anonymizing content that often relies on cited prior work.
You're the second person to bring this up, but I'm not sure why it's a problem. Just don't say "based on my own previous work" and instead say "based on previous work". I.e. cite yourself the same way that you'd cite anyone else's work.
Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study
#144Very common for anyone who has ever spent time in academia. When I was still in grad school I noticed a trend in some major mathematical and CS journals where a lot of time it would be the same N people writing articles. This was driven home when my advisor told me "without such and such's name this paper has no chance of making it past review". Non-sense. However, for all the people that say "trust the science" this…
You mean like Elon Musk? When he came up with that giant box to extract kids from a cave with passages so narrow rescuers couldn't wear their air tanks on them, they had to push them ahead of themselves, reddit and HN and twitter were full of people defending him. There were even people defending him when he accused the rescue leader of being a pedo - simply because he was british and living in thailand.
Or how about the CEO of Brave, who apparently is an expert in public health, viruses, vaccines, epidemiology, etc? Endlessly defended here on HN any time his name comes up.
Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study
#145Earlier quoted context omitted.
There was a study some time back where they set folks up in groups, then had them listen and vote on bands / music. Each group trended towards catapulting a single band / musician, but it always just depended on which one in the group got the momentum first. I wish I could find the study again but it's hard to google for. Pretty eye opening.
Salganik MJ, Dodds S, Watts DJ Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market https://www.science.org/doi/10.1126/science.1121066
Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study
#146I'm not sure what we're supposed to find astonishing about academics (like most of humanity) being vulnerable to reputation biases in their reasoning. As in, that would be the null hypothesis. It would be astonishing if most academics overcame it. EDIT: apparently, the size of the effect (factor of six more likely to be accepted if it came from a Nobel Prizewinner) is larger than anticipated.
Are people in general bias based on reputation?
Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study
#147It blows my mind that peer reviews aren't done blind. Not only is there this geneder bias but also bias of "I know that person they're famous I'm sure their work is good" or "I've rejected their paper in the past". Seems like an easy fix would be to make all peer reviews blind. In fact, for any study that gets federal funding, they should have to publish their hypothesis ahead of time into an escrow system, submit th…
These issues? Off the top of my head:
1. Quantity over quality
2. Normalised sensationalising of ones research
3. Neglecting good or even necessary collective scientific practices such as replication studies and data and code sharing and openness.
a. Here, valuing the actual work of peer review comes in as a fundamental aspect of the scientific process that should be respected, rewarded and published rather than being some silent aristocratic duty that eventually gets navigated around in the sensationalism rat race many researchers pursue.
4. Selecting for "impact" and "novelty" and not the quality of a researcher's/scientist's administrative and leadership skills, scientific method and integrity and teaching skills (incl, importantly, the teaching of graduate students) a. Though novelty and impact are important in research, IMO, they're outcomes that are hard/impossible to select for largely because research breakthroughs are often serendipitous and the kind of true genius that will "hit targets no one else can see" is rare and frankly everyone knows it when they see it (provided they're good researchers and not just good salespeople). A very senior academic once told me in a private context that you can't predict where the breakthroughs are going to come from as an inexperienced researcher playing around is just as likely to make a breakthrough as a senior researcher with many credentials.
5. A personal belief of mine ... resisting the necessary professionalisation of research, by which I mean that the conception of a researcher is based very much in the same model we had centuries ago. IE, a lone genius researcher left to their own devices to find the truth. Scientific research, at least, is now too complex, too hard and involved for this to be true. Collaboration and more and more specific roles are necessary to make the industry work well. The resistance of the "industry" to recognising the importance of software developers in research is a perfect example. Same goes for statisticians and consultants from adjacent fields. My personal favourite for such an "associative" role would be quasi-theoreticians (outside of physics) who can aid in aggregating, reviewing and critiquing the literature in real time without necessarily having a horse in the race.~~~
Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study
#148Earlier quoted context omitted.
First of all, Knuth would never publish something "garbage" because his reputation would be on the line for everyone to read (forever).
Or he would never publish something "garbage" because people are convinced by his reputation that it will not be garbage. Emperor's new clothes, while the rest need to be whiter than white and all that. These things happen because people are blinded by titles in a complex world with complex people.
Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study
#149Earlier quoted context omitted.
Peer reviews are usually blind (authors don’t know who their reviewers are). What you are referring to is double blind (reviewers don’t know who the authors are). The challenge with double blind is anonymizing content that often relies on cited prior work.
> The challenge with double blind is anonymizing content that often relies on cited prior work. You're the second person to bring this up, but I'm not sure why it's a problem. Just don't say "based on my own previous work" and instead say "based on previous work". I.e. cite yourself the same way that you'd cite anyone else's work.
Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study
#150Earlier quoted context omitted.
First of all, Knuth would never publish something "garbage" because his reputation would be on the line for everyone to read (forever).
I mean it was an example I chose because the name recognition in HN. I would doubt Knuth would publish garbage. Then again, how would you know? If you've never been in academia it's understandable to think that way. But if you have, you realize the immense pressure, self-doubt, and scrutiny you'd have even making a bold, well researched, attempt to prove someone like knuth wrong even if it was obvious he was. I recal…
Rather, Knuth could publish some mediocre paper that would be instantly accepted by all the top journals, widely read, discussed and cited instead of more salient works on similar topics.
That's the problem really, excellence is rare while mediocrity is aplenty and academia is a numbers game. The mediocres with prodigious output will get the cites, tenure, TAs, lab assistants and budgets to do research and perhaps stumble onto some other mediocre result, reinforcing the cycle.