AI intensifies fight against ‘paper mills’ that churn out fake research

Journals unfortunately aren't stepping up to meet the challenge. I was writing articles about this problem several years ago and things haven't noticeably improved. This thread is full of people pointing out the obvious solution - take claim audit seriously and start by paying people to do professional peer reviews. Their actual solutions tend to look like spam filters for paper submission queues. It's enough to be able to say they're doing something, but not enough to actually make a big difference especially post ChatGPT.

I'm not so pessimistic, I think people learn to discriminate between sources. A lot of people right now are learning to generalize to "experts aren't" but it's a rather more nuanced understanding than the media like to make out when you dig in, for instance people understand that "expert" in this context usually means public sector funded academic or civil servant, and not e.g. a roughneck on an oil rig, some UI programmer at Apple, the guy who fixed their car last weekend. They learn that the first form of expertise is the type where making false claims can be beneficial for the people making them, whereas if you lie a lot about oil on an oil rig eventually something will explode.

I think we'll eventually get to a place where incentives are better aligned, for instance where data collection and aggregation is fully divorced from the people analyzing it. A lot of the distortion in science comes from the fact that academics need to collect data but aren't rewarded much for doing it, so once a dataset is collected it gets kept secret and that allows for a lot of dishonest game playing. It also means people are strongly incentivized to see things in the data that aren't really there. If data collection and analysis was fully divorced, those issues would go away (you'd get other problems of course but would they be worse?).

The contrast between ML and physics is equally interesting, and seems to corroborate the experience of another poster, who expresses doubt about removing peer-review from long-haul studies in biological sciences. I’m still struggling to put a finger on why exactly this difference might exist, though.

Anyway, thanks for your thoughts :)

For the difference between ML and physics/other sciences, there are a few points that I see.

- First, ML submits to conferences rather than journals. This means typically you have one shot at acceptance, and possibly one chance to rebuttal. Other fields use journals, where there often is a push to accept and that the reviewer's job is to make the paper better rather than filter absolutely. This results in no one actually advocating for the paper to be published other than the authors (no incentive to accept). Worse, we have a zero sum game, so if reviewers are also authors of other works (common) they have an optimal strategy of rejecting other players' works (large incentive to reject).

- Second, ML is extremely hype driven. The field just moves lightning fast, it is insane. This leads to incremental improvements (technically true for all science) but muddies the waters since it is easy to reject on low novelty but if you don't release your work fast you get scooped and get rejected for again low novelty. Unfortunately we don't define concurrent work well (and I've had works actually rejected because we did not cite works that were publicly released (arxiv) around the same time as us or, worse, after we submitted to the conference!).

- Third, there are a large number of people submitting works (nearly 10k per conference), the field is wide (reducing domain expertise), and highly empirical (few learn theory and base understandings and instead rely on benchmark results to evaluate). The size brings with it a lot of noise and the empirical nature reduces rigor (noise in evaluation). The large number of works also results in low reviewer quality as chairs scramble to add reviewers to works (in my last review, 3 of 4 reviewers self reported 3/5 confidence scores, meaning they did not work in this domain). So we just have way more sources of noise than would be seen in many other fields. Consistency experiments support this as well: finding reviewers are good at rejecting bad works but bad at rejecting good works. Which concludes that reviewers over reject (default strategy).

I also appreciate your opinions. I certainly don't have all the answers. As I see it, we got to decide as a community. The only way to do that is to hear complaints, proposals, and challenges. We're all in this together so to me it is most important that we openly discuss these. Thank you for your thoughts and comments.