Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI intensifies fight against ‘paper mills’ that churn out fake research

nature.com

Post reply on HN
Print view View original

31–40 of 186 posts

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#31

Post by SQueeeeeL »

ethanbond wrote: post #27
huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

The cases you mentioned have clear and immediate commercial incentives that push them toward quality. Not quite so for basic research that might (optimistically) be decades away from commercialization and will probably be commercialized from someone very very far away in the value chain from the original researcher. FWIW I'm also really critical of peer review/journals, just don't think this is a great line of analog…

I've found most people who are deeply critical of peer review as a process (and not just how shady modern journals are) have never engaged in serious scientific work. They have no idea how lost in the weeds researchers can get, and how useful having a somewhat uninformed 3rd party see if they can parse your prose is for having a work be understandable. Commercialization really only applies to a very small subset of scientific discoveries

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#32

Post by webnrrd2k »

huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

Re: "Science does not get worse when there are more bad papers"...

I don't think that this is true at all. Weeding through bad papers is, at a minimum, an opportunity cost, as is a good paper built on top of a bad one.

Also, there is a societal cost in that bad research can get picked up and believed by people, like the anti-vax crowd. Or, bad research can be used to push an agenda, like anti-climate change.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#33

Post by LadyCailin »

It feels like AI is exposing Kessler Syndrome for other areas - that is, some small scale amount of junk isn’t a problem necessarily, but if you scale up that problem, it fundamentally changes the thing for everyone permanently. I guess the jury is still out whether it’s net good or bad, but it feels like it’s going to force us to confront some fundamental issues we have in society, which have perhaps always been there, but are now unavoidable, and will demand quick societal change. That almost never goes over well.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#34

Post by pierat »

worrycue wrote: post #30
pierat wrote: post #21

Then, lets use the 'AI' as its own proving ground. "AI, here's the text of this paper. We need to determine if it's accurate and true. It may be a false paper, and our goal is to determine that. AI, please analyze for logical fallacies, and list and cite them here. Next, for the potentially falsifiable claims, please give me a list of potential experiments that I can do or verify to check the claims being made." Edit…

I don’t understand why you would think current AI can do any of that. Current AI can’t even stop hallucinating.

Because I was sub'ed to openAI until yesterday. And my experiments with it were pretty promising.

I took the whole corpus of Magic the Gathering rules ( https://magic.wizards.com/en/rules ) as a textfile, and fed it into 4.0 . parsed it in a few seconds. I was then able to send it cards from Gatherer (MtG card database), and then ask it pointed questions about multiple card interactions.

I also compared it to what DCI judges have made ruling on as well, and matched 100%. It quite impressed me.

I was thinking next is to give it the rules text, and a JSON of every card. And then ask for all combos. But I'd run out of response before it could.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#35

Post by lacker »

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well.

It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results. It's common to use things like manual Jupyter notebooks, unlabeled CSVs, and a bunch of disorganized data files, in a one-off process that a scientist manually summarizes to produce a paper.

To me, in an ideal world each paper would sit in a GitHub repository, with an integration test that verifies the code actually produces the results used in the paper. That isn't really what academics prioritize, but perhaps things will move in this direction as more people realize that we have a replication crisis, and also as scientists tend to have more software engineering skills over time.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#36

Post by jfengel »

noughtme wrote: post #25
EGreg wrote: post #12

Earlier quoted context omitted.

Guns don’t kill people. They have no agency. But they sure make it easier.

Illegal guns kill people. Something like 80-90% of guns involved in homicide are illegally obtained/possessed. This is relevant because like banning guns, “banning” AI shouldn’t be the focus. It seems like an AI detection arms race is inevitable.

Illegal guns start as legal guns. With both AI and guns you need to be willing to think about the problem at every step in the chain, not just at the end.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#37

Post by photochemsyn »

The Jan Hendrik Schon scandal of two decades ago and the fallout from it point to how to limit the spread of fraudulent research into the accepted literature:

https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal

The basic issue is that all raw data must be preserved and made available for scrutiny by other researchers after publication, as must experimental materials. Why?

> "The committee requested copies of the raw data, but found that Schön had kept no laboratory notebooks. His raw data files had been erased from his computer. According to Schön, the files were erased because his computer had limited hard drive space. In addition, all of his experimental samples had been discarded or damaged beyond repair."

The opposition to this standard is strongest in the corporatized patent-centric research sectors, which is most of applied science in the USA and China, etc. Ambitious academics in non-commercial sectors don't really like it either as it means competitors can jump-start their research by having access to their raw data and experimental protocols. Regardless, implementing standard practices with respect to laboratory notebooks, raw data, and experimental materials in any institution receiving federal research money makes a lot of sense - along with regular audits, with failure leading to a cutoff in funding.

This problem is much broader than just the paper-mill outfits the article focuses on; some highly public and contentious related issues are public access to the raw data, research records and database sequences from the Wuhan Institute of Virology from c.2016-2019, the raw clinical trial data from Pfizer/Moderna/J&J Covid vaccine trials in 2020, and so on.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#38

Post by CobrastanJorji »

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

Imagine a world where all the raw data sits in a public repository next to a script to process the data which was reviewed as part of the publication process, which would accompany the paper and could be quickly and easily replicated with new data. Anyone could produce the graphs shown in the paper simply by downloading both and running them together. What a wonderful thing that would be. And you could imagine the government funding the storing and serving of all the data. What a wonderful daydream of an idea.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#39

Post by ushtaritk421 »

EGreg wrote: post #20
ushtaritk421 wrote: post #16

Feels like AI makes the fight against paper mills easier. A group could submit AI-generated papers to different places and create a public index based on how successful they are at getting their nonsense papers accepted.

Your comment seems self-contradictory. AI makes the fight against spam harder since it is harder to detect fakes. The group would only get more successful over time. The “AI detection” tools get worse over time and overhyped already anyway, as we learned from the guys running that sci fi submission site.

Right, but an org that accepts fake papers (now easily generated) gets outed easily and ignored by everyone.

Legitimate orgs with real review processes presumably sniff out the fake papers easily with their current procedures and don't publish them.

And I guess if AI gets good enough that an expert reviewer learns something novel by reading it then it deserves to get published.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#40

Post by jfengel »

pierat wrote: post #34
worrycue wrote: post #30

Earlier quoted context omitted.

I don’t understand why you would think current AI can do any of that. Current AI can’t even stop hallucinating.

Because I was sub'ed to openAI until yesterday. And my experiments with it were pretty promising. I took the whole corpus of Magic the Gathering rules ( https://magic.wizards.com/en/rules ) as a textfile, and fed it into 4.0 . parsed it in a few seconds. I was then able to send it cards from Gatherer (MtG card database), and then ask it pointed questions about multiple card interactions. I also compared it to what DC…

Out of curiosity, did you try the same experiment without specifically training it on the MtG rules? Could it have all of that, including card interaction decisions, from its training data (sourced from the whole Internet)?
Post reply on HN