Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Authors’ names have ‘astonishing’ influence on peer reviewers: study

nature.com

Post reply on HN
Print view View original

301–310 of 319 posts

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#301

Post by TheRealPomax »

Null-Set wrote: post #36
TheRealPomax wrote: post #27

Earlier quoted context omitted.

This is literally how one of the biggest scandals in academic publishing occurred[1], so... no? Maybe it's a nice way to treat experts when asking them for their opinion, they are experts after all, but we absolutely don't want to give celebrities more leeway than unknowns in research papers where new claims are being made. You better have done the work properly, again, and again, and again. No free passes, no blind…

In this case it looks like his good reputation was falsified from the start? They were unable to prove misconduct in his dissertation only because the data had been destroyed. So consistently missing his misconduct would be more to blame here than riding on a reputation.

But that ignores the years prior where he had enough reputation to be scrutinized less, which lets him publish more, which made him more famous, which got him scrutinized less, allowing him to keep publishing, boosting his reputation, [etc].

Right up to the very end, this man was able to publish like a celebrity because no on questioned it because he was famous. So let's not bake that into the process: scrutinize all papers equally. Experts don't get a free pass, if they have new claims to make, those claims are just as "I don't believe you yet" as anyone else's.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#302

Post by CodeSgt »

Test0129 wrote: post #298
CodeSgt wrote: post #128

Earlier quoted context omitted.

There's a lot of middle ground between blindly trusting all 'science' and blatantly disregarding everything that rubs you the wrong way. It's possible to both have a generally high amount of faith in 'the science' while still remaining critical of its flaws, or rather, the flaws of the system in which its conducted.

I don't think this is the case. The point of science as one commenter pointed out is a lack of trust (or faith). Hence the need for pages and pages of sources, concerted efforts on reproducibility, etc. In certain fields reproducibility is extremely important and a single scientific article should be taken with extreme skepticism until it's reproduced several times. I think this is the thing that's often skipped. A s…

Trusting the outcomes produced by science is inherently faith for most people who aren't an expert in that particular field of research, and since no one is an expert in every field everyone has to take certain scientific outcomes on faith.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#303

Post by caddemon »

ljw1001 wrote: post #268
caddemon wrote: post #264

Earlier quoted context omitted.

That would be a good idea, and then have the names revealed in a future version like the way the paper itself can be updated. I don't think it is possible though. If it is, it certainly isn't used.

Without peer review, publications without names would be untrusted and untrustworthy, and would probably represent a security issue

The main point of arXiv is not to substitute for actual publication. Some people use it that way, which could still occur. But most intend to submit their work for review to a venue with a supposed double blind process. The use of arXiv is for flag planting and ease of open access.

Often the arXiv upload will be right around the time of a conference deadline in fact. For these people I see no problem with an OpenReview-esque setup where the arXiv page remains anonymous until the authors opt to reveal themselves.

If people don't care for true double blind review then they should stop pretending they do. Conferences like ICLR and NeurIPS are far from blind. It would be one thing if it were just a matter of trusting reviewers to not Google the paper title, but it goes beyond that. There are currently no restrictions on the use of social media to advertise works under review nor on the timing of arXiv uploads, which can trigger alerts to those in the field right around review time.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#304

Post by seanmcdirmid »

jedberg wrote: post #143
seanmcdirmid wrote: post #139

Earlier quoted context omitted.

Peer reviews are usually blind (authors don’t know who their reviewers are). What you are referring to is double blind (reviewers don’t know who the authors are). The challenge with double blind is anonymizing content that often relies on cited prior work.

> The challenge with double blind is anonymizing content that often relies on cited prior work. You're the second person to bring this up, but I'm not sure why it's a problem. Just don't say "based on my own previous work" and instead say "based on previous work". I.e. cite yourself the same way that you'd cite anyone else's work.

Speaking from personal experience, it can be really difficult hard to write a paper that anonymizes yourself...especially if you are known for a particular approach to a specialized problem. You actually have to begin excluding cites, turning them into placeholders (withheld due to anonymous review requirements), rather than just speaking about the work as if someone else did it.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#305

Post by 9wzYQbTYsAIc »

lupire wrote: post #258
9wzYQbTYsAIc wrote: post #176

Earlier quoted context omitted.

When I was studying undergrad psych, Psychology Today was falling into disrepute. Are they still a reputable source, today?

It can't be "still reputable" after "falling into disrepute".

“was falling” in the imperfect tense.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#306

Post by gms7777 »

hannob wrote: post #134
jedberg wrote: post #121

It blows my mind that peer reviews aren't done blind. Not only is there this geneder bias but also bias of "I know that person they're famous I'm sure their work is good" or "I've rejected their paper in the past". Seems like an easy fix would be to make all peer reviews blind. In fact, for any study that gets federal funding, they should have to publish their hypothesis ahead of time into an escrow system, submit th…

> It blows my mind that peer reviews aren't done blind. I've co-authored a few papers and blind reviews have some surprising consequences, like discussions along the lines of "you can't say that you already wrote about this issue elsewhere and put a link in the paper, because that would unblind it". I was a bit uncomfortable with that, because I like to "cite my sources" (even if the source was myself in this case).…

Another factor in my field as well is linking to software -- papers that introduce algorithms or use computational analyses are rightfully expected to include the code (usually github) and reviewers are expected to at least check that the code exists and is reasonably documented. I can remove my name from the paper, but the github page will say which lab it's coming from.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#307

Post by gms7777 »

hardtke wrote: post #25

Fairness and efficiency are often at odds. As a reviewer, I was sometimes asked to review papers from unknown authors at unknown institutions. My level of effort was likely proportional to how well I knew the quality of the institution from which the paper came. There is a great deal of time involved in understanding a paper well enough to do a full review.

I agree. I'm in an applied computational field, that attracts a lot of computational and mathematical researchers that don't have knowledge of the applied system. I've learned from experience that often researchers new to the field make a whole lot of poor assumptions and modeling decisions, even if the mathematical part is fine. For better or worse, I'm going to more heavily scrutinize the details of a paper from a lab I haven't heard than I am one that I believe has a strong track record. That doesn't mean I'll give a pass for the latter, I'll still read it with a critical eye. But certain types of trust build up over time.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#308

Post by mturmon »

eldenwrong wrote: post #182
geysersam wrote: post #170

Earlier quoted context omitted.

Agree. It's stagnant, fearful and bureaucratic. "Trust the science" never changed anyones mind. However, I think your climate change example is a bit strange. If anything, it's big oil who's been trying to sell the idea that we don't have to stop using fossile fuels. Hiding evidence, and spreading confusion by paying lobbyists and scientists. Global warming was proved beyond reasonable doubt decades ago.

Climate change is one of the most politicized fields at the moment. Any questioning it, even slightly means being banned from grants and academia. Its also interesting that most climate models are NOT open source. Most recorded data from satellites is also NOT open source. So everyone works with a pre cleaned data set. Its also worth pointing out that data sets like HadCRUT have never been audited by any respected sc…

> Most recorded data from satellites is also NOT open source.

This is false. NASA and ESA science data is free.

There is often an embargo period for very novel sensors, and always a delay of hours-to-weeks to allow processing to catch up, but it's free.

If it's the source code of the analysis pipeline you mean -- even though you said data - that's a harder lift, because the processing is complex. But even that is changing (https://science.nasa.gov/open-science-overview).

Even in the absence of the open science initiative above, today you can always get the raw data ("Level 1 radiances") or sometimes even uncalibrated straight-off-the-sensor data ("Level 0"), if you want to process it. (https://www.earthdata.nasa.gov/engage/open-data-services-and... -- "All EOS instruments must have Level 1 Standard Data Products (SDPs)")

And if you want to look in to how the processing works, there are detailed documents ("ATBD's") that explain how the pipeline works, for each data product. Also free.

> Climate scientists are not really data experts, ....

Dreadfully wrong. Do you work in this area at all?

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#309

Post by goto11 »

nemo44x wrote: post #122
emacsen wrote: post #94

When I was an undergrad in college, I helped design a study on the role of gender perception in expertise. We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials. We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's…

It’s one thing to observe a pattern in a certain subset of people. However, why do you think there is this apparent bias or perception? If there is a bias, is it good, bad, or neutral? Are there biases in other directions?

The example described is the textbook definition of bias. Given the exact same text, the participants rate it higher if a male name is attached.

Re: Authors’ names have ‘astonishing’ influence on peer reviewers: study

#310

Post by anm89 »

themitigating wrote: post #281
anm89 wrote: post #154

Earlier quoted context omitted.

Much of academia seems to have become an outwardly transparent no hold barred status battleground. So this just seems to exactly align with my expected outcome given that assumption. I'm not saying there aren't people in academia who aren't driven by doing good research but I certainly don't see that as the driving force in the US academia.

Do you have proof of this?

Yes I've got objective proof of my qualitative opinions on sociological topics. Let me just dig that right up for you.

I didn't realize that HN was an academic journal where all statements should come with proof.

Do you have proof that I'm wrong?

Post reply on HN