Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI intensifies fight against ‘paper mills’ that churn out fake research

nature.com

Post reply on HN
Print view View original

81–90 of 186 posts

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#81

Post by lumost »

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

The way academic research is setup is not currently conducive to effective research. Research must intrinsically be allowed to fail. If you incentivize success, then what happens when someone is 2-4 years in and their only skill is apparently worthless?

Without Tenure or the ability for a scientist to accumulate their own savings, there is a strong incentive for most scientists to charitably interpret results/papers to maintain relevance.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#82

Post by stubybubs »

godelski wrote: post #63
omginternets wrote: post #57

Prediction: as a result of fake research and it’s ilk (fake credentials, fake trade-journal pubs, fake reviews, etc.), we will observe a renewed interest in referrals and meatspace networking. It will become correspondingly difficult for outsiders to enter professional circles.

On the other hand, maybe we'll abandon peer review and return to a pre 1950's like style? Everyone in ML just uses arxiv because the field is moving so fast. All work is realistically peer reviewed once it is out there. We treat conferences (more important than journals) as very noisy signals. Unfortunately, we still use those as metrics for completing degrees or hiring people. The way I would try to make this better…

It's easy to do this with anything in computer science as others can implement or recreate whatever is being discussed easily. Not so with a 6 month experiment with several groups of humans, or even a 2 week one with rats.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#83

Post by juve1996 »

burkaman wrote: post #8
naillo wrote: post #6

The problem is the people doing this. The AI doesn't have agency, it's just an autocomplete tool.

That doesn't really seem relevant to the discussion. AI obviously makes it 1000x easier to generate realistic-looking fake papers, so it "intensifies" the issue as this article explains. It is newsworthy and discussion-worthy, and as you said AI doesn't have agency so we don't need to worry about hurting its feelings by referencing it in a headline.

The only reason it's a problem is that papers became the measure - more papers = more prestige. The measure will simply change because now it won't have a strong signal.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#84

Post by noobermin »

godelski wrote: post #60

There's really only one solution to this. Luckily it is the same solution to a lot of other things. Unluckily it is the same solution that no one has sought to implement for decades. Reviewing takes a lot of nuance, care, and time. You're not going to get this when all the incentives for the reviewers are currently to reject works. You can only get a group to work on ethics alone when that group is small and accounta…

1) You must therefore pay the reviewers. Probably a good starting point.

2) It is in fact, hard to publish things already in top journals especially if you're not established. Given the fact you also want the pool of reviewers to shrink, you're also making difficulty of publishing works even harder. Yes, the issue is paper mills, got it, but you're making it even harder for sincere scientists who aren't part of a mill in doing this.

3) The flood of work that will fall to the smaller, more accountable group will require culling, which of course, the top journals will cull the horde based on the biases they already have: pedigree, which will further calcify the issues in science and research already, or even intensify it.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#85

Post by mycologos »

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

It might help to expand on "bogus". Bogus has a few levels, going from "not good, but possible from a well-intentioned author trying to do the right thing" (low-level bogus) to "outright deception" (high-level bogus). Small sample sizes, statistical errors, and flawed (but honest) experiment design are all, I suggest, low-level bogus. Faking data and plagiarism are high-level bogus.

I think peer review is capable of, eventually, mitigating low-level bogus. The quantitative standards in fields where low-level bogus is a problem (e.g., but definitely not only, medicine) are rising. Peer review is not a scalable solution to high-level bogus. Figuring out high-level bogus seems to be almost a full-time job [1]. You cannot expect this level of effort from researchers, especially if they are reviewing for free; I would even argue that it's easier to fake data than to figure out it's fake. It also requires more expertise to assess quality research than to write a low-level bogus paper and submit it. There's a mismatch here. There are not enough expert reviewers to handle all the low-level bogus papers.

The solution therefore seems to require some kind of reputational component. There needs to be a cost to engaging in high-level bogus. But this is a hard problem. Do you ban any lead author of a paper with demonstrated high-level bogus? Publicize their names? Ban any author of a paper with demonstrated high-level bogus? Throttle the submissions any one person can make to a conference/journal at a time? I don't know. But the current model will have to change.

[1] https://www.ft.com/content/32440f74-7804-4637-a662-6cdc8f3fb...

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#86

Post by noobermin »

SpaceBuddha wrote: post #53
bagels wrote: post #22

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Sokal_affair

"Later, after Sokal revealed the hoax in Lingua Franca, Social Text's editors wrote that they had requested editorial changes that Sokal refused to make, and had had concerns about the quality of the writing: 'We requested him (a) to excise a good deal of the philosophical speculation and (b) to excise most of his footnotes.' Still, despite calling Sokal a 'difficult, uncooperative author", and noting that such write…

That even makes it worse, I somehow didn't know this part of it.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#87

Post by yifanl »

samstave wrote: post #24
captainbland wrote: post #13

Earlier quoted context omitted.

This seems like a bleak dystopian fiction concept actually. AIs being in control of human experimentation on the basis of well meaning but poorly interpreted ethical guidelines (with, of course, fully hallucinated loopholes). Maybe add in that this has become one of the few sources of income for people as more and more jobs have been lost to the machines so many are compelled to participate.

" Take all the research done in [field] and compare the papers written by N authors, and look into their citations and compare the outcomes of what was published and find where the summation of the reads of their papers as shared citations and summarize the research where they agree, disagree and compare with international organizations and their papers coming from specifically the leads in this field from countries…

Physically repeating the experiments and independently verifying the data in the research is missing.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#89

Post by dmingod666 »

SiempreViernes wrote: post #42
dmingod666 wrote: post #10

Not even a mention of the galactica model. I suspect the author probably isn't aware of its existence..

To be fair, it is stressful to keep track of all the faceplants the new saviours perform

Fair enough, but this was a major 'event' in both tech and academic circles..

Funny that Nature publishes an article about fake papers and fails to mention the one model specifically trained exclusively on white paper to write (fake?) white papers..

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#90

Post by mschuster91 »

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

> Most scientists cannot even reproduce their own results. It's common to use things like manual Jupyter notebooks, unlabeled CSVs, and a bunch of disorganized data files, in a one-off process that a scientist manually summarizes to produce a paper.

The only way to get that under control is if universities had a career track of data scientists and programmers, basically a shared resource pool of specialists that all researchers could use. But most US universities seem to prefer to "invest" their endowments into professional athlete teams.

Post reply on HN