Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI intensifies fight against ‘paper mills’ that churn out fake research

nature.com

Post reply on HN
Print view View original

21–30 of 186 posts

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#21

Post by pierat »

Then, lets use the 'AI' as its own proving ground.

"AI, here's the text of this paper. We need to determine if it's accurate and true. It may be a false paper, and our goal is to determine that.

AI, please analyze for logical fallacies, and list and cite them here.

Next, for the potentially falsifiable claims, please give me a list of potential experiments that I can do or verify to check the claims being made."

Edit: Wow, at -2, and attracting all the haters. Thought this place was about inquisitiveness and curiosity about tech. Oh well.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#22

Post by bagels »

ushtaritk421 wrote: post #16

Feels like AI makes the fight against paper mills easier. A group could submit AI-generated papers to different places and create a public index based on how successful they are at getting their nonsense papers accepted.

https://en.wikipedia.org/wiki/Sokal_affair

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#23

Post by logifail »

tikkun wrote: post #3

The upside from this downside is that it increases the performance of papers that can be replicated easily. In fact, once AI can replicate papers, that'll be helpful for verifying a lot of research.

> once AI can replicate papers

Is this AI to be provided with a group of poorly-remunerated postgrads and postdocs and a laboratory full of relevant equipment in which they may conduct experiments under its direction?

(Full disclosure: have a couple of very minor published papers obtained under the non-AI version of this exact system...)

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#24

Post by samstave »

captainbland wrote: post #13
nradov wrote: post #9

Earlier quoted context omitted.

How will AI be able to replicate papers?

This seems like a bleak dystopian fiction concept actually. AIs being in control of human experimentation on the basis of well meaning but poorly interpreted ethical guidelines (with, of course, fully hallucinated loopholes). Maybe add in that this has become one of the few sources of income for people as more and more jobs have been lost to the machines so many are compelled to participate.

"Take all the research done in [field] and compare the papers written by N authors, and look into their citations and compare the outcomes of what was published and find where the summation of the reads of their papers as shared citations and summarize the research where they agree, disagree and compare with international organizations and their papers coming from specifically the leads in this field from countries X,Y,Z"

Create a paper based on this information and coalesce all this data into a new outcome. Do not Lie or make up data. list where you think you are lacking in data - or based on searching, which datasets or companies should be included in the list. Create a table for all the citations sourced, with links and a comment why it is or is not included in your findings"

EDIT: Someone down voted this, and I am learning 'Prompt Engineering' - can someone ELI5 why this is a bad prompt? Seriously, can someone explain what sucks about the question?

Is it a stupid premise or is the crafting of the prompt lame?

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#25

Post by noughtme »

EGreg wrote: post #12
naillo wrote: post #6

The problem is the people doing this. The AI doesn't have agency, it's just an autocomplete tool.

Guns don’t kill people. They have no agency. But they sure make it easier.

Illegal guns kill people. Something like 80-90% of guns involved in homicide are illegally obtained/possessed. This is relevant because like banning guns, “banning” AI shouldn’t be the focus. It seems like an AI detection arms race is inevitable.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#27

Post by ethanbond »

huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

The cases you mentioned have clear and immediate commercial incentives that push them toward quality. Not quite so for basic research that might (optimistically) be decades away from commercialization and will probably be commercialized from someone very very far away in the value chain from the original researcher.

FWIW I'm also really critical of peer review/journals, just don't think this is a great line of analogy.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#28

Post by dsr_ »

nradov wrote: post #9
tikkun wrote: post #3

The upside from this downside is that it increases the performance of papers that can be replicated easily. In fact, once AI can replicate papers, that'll be helpful for verifying a lot of research.

How will AI be able to replicate papers?

The same way a Xerox machine does, but with different wording and randomly synthesized sources.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#29

Post by sebzim4500 »

riku_iki wrote: post #17
api wrote: post #2

You get what you incentivize. We incentivize papers so we get papers. Also anything used as a metric tends to cease to be a good metric.

There is a good metric in AI research: reproducible results on well known benchmarks (Big Bench for example).

There's no good benchmark that tests for most of the issues with LLMs, for example.

Maybe if we had a benchmark that told you how often a model hallucinated we would have solved the problem by now.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#30

Post by worrycue »

pierat wrote: post #21

Then, lets use the 'AI' as its own proving ground. "AI, here's the text of this paper. We need to determine if it's accurate and true. It may be a false paper, and our goal is to determine that. AI, please analyze for logical fallacies, and list and cite them here. Next, for the potentially falsifiable claims, please give me a list of potential experiments that I can do or verify to check the claims being made." Edit…

I don’t understand why you would think current AI can do any of that. Current AI can’t even stop hallucinating.
Post reply on HN