Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI intensifies fight against ‘paper mills’ that churn out fake research

nature.com

Post reply on HN
Print view View original

41–50 of 186 posts

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#41

Post by EGreg »

noughtme wrote: post #25
EGreg wrote: post #12

Earlier quoted context omitted.

Guns don’t kill people. They have no agency. But they sure make it easier.

Illegal guns kill people. Something like 80-90% of guns involved in homicide are illegally obtained/possessed. This is relevant because like banning guns, “banning” AI shouldn’t be the focus. It seems like an AI detection arms race is inevitable.

Wait a moment. If 80-90% of guns used to kill people are illegally obtained, that means the laws successfully generate a regime by which guns are obtained much more safely. Now all that remains is to enforce the laws and crack down on illegal gun sales (although in USA that isn’t all since old guns already purchased illegally will continue to work for decades)

It’s like when people were saying that 80-90% of people dying in hospitals were unvaccinated. So vaccination helped right?

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#43

Post by artursapek »

huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

I forget which episode, but Andrew Huberman discussed somewhere how the incentives behind getting published in science journals and the peer review process often lead to poor studies.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#44

Post by EGreg »

ushtaritk421 wrote: post #39
EGreg wrote: post #20

Earlier quoted context omitted.

Your comment seems self-contradictory. AI makes the fight against spam harder since it is harder to detect fakes. The group would only get more successful over time. The “AI detection” tools get worse over time and overhyped already anyway, as we learned from the guys running that sci fi submission site.

Right, but an org that accepts fake papers (now easily generated) gets outed easily and ignored by everyone. Legitimate orgs with real review processes presumably sniff out the fake papers easily with their current procedures and don't publish them. And I guess if AI gets good enough that an expert reviewer learns something novel by reading it then it deserves to get published.

That’s exactly the point. What you previously assumed review processes could catch is no longer true. AI can now infiltrate pretty much anything, especially if there is a Generative Adversarial Network (which “AI detection tools” can be used to train at scale).

And if you say it “deserves to get published” then taken to its logical conclusion, human generated content in all fields and interactions will soon be dwarfed by AI content and interactions.

The issue is that the AI can be switched out after it’s infiltrated and taken over, or it can be gradually used to shift public opinion or organize any sort of coordinated attack. Heck, a reputational attack is easy to pull off at scale within 6 months via AutoGPT already, and it takes 1 button press.

First they’ll separate us, then they’ll herd us into echo chabers and cause our protests won’t be heard by anyone anymore amid all the AI glut.

It comes gradually, then all at once!

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#45

Post by habitzreuter »

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

(PhD student in STEM here) I think most people have the wrong idea about what peer review is. My advisor teaches us to treat it as a first check, but it is not a guarantee of correct results.

Most of the time I spend on research is actually trying to understand the literature and reproducing their results. If I can't do it, it probably means I don't understand enough about the work I'm reading, but there is also the small chance that the published analysis is wrong, which already happened to me.

EDIT: typo.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#47

Post by raincole »

pierat wrote: post #34
worrycue wrote: post #30

Earlier quoted context omitted.

I don’t understand why you would think current AI can do any of that. Current AI can’t even stop hallucinating.

Because I was sub'ed to openAI until yesterday. And my experiments with it were pretty promising. I took the whole corpus of Magic the Gathering rules ( https://magic.wizards.com/en/rules ) as a textfile, and fed it into 4.0 . parsed it in a few seconds. I was then able to send it cards from Gatherer (MtG card database), and then ask it pointed questions about multiple card interactions. I also compared it to what DC…

> But I'd run out of response before it could.

So you used ChatGPT via the website.

> I took the whole corpus of Magic the Gathering rules ( https://magic.wizards.com/en/rules ) as a textfile, and fed it into 4.0 . parsed it in a few seconds.

You mean the TXT file? ChatGPT(4) on the website literally can't comprehend it. It has much, much, much, much more tokens than ChatGPT(4) can take.

So this particular example proved what your parent comment pointed out. You think AI can do something that it can't.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#48

Post by nologic01 »

This is No.2 in the list of existential AI risks [1]

> A deluge of AI-generated misinformation and persuasive content could make society less-equipped to handle important challenges of our time.

One way to think about it as significant chunks of information exchange turning into a Market for Lemons. Namely the information asymmetry between the producer of AI junk and the receiver of said junk means that the receiver cannot distinguish between a high-quality message (a "peach") and a zero (or negative) value "lemon". Then receivers are only willing to pay a fixed price for a message that averages the value of a "peach" and "lemon". Given the zero marginal cost of producing junk, this will mean that in the limit receivers will be willing to pay exactly zero. Information exchange is completely discredited.

But is this really an "existential risk" or an opportunity to think deeply about human relations, trust and the meaning of exchange?

Maybe the transactional, "a fool is born every second", buyer beware, caveat emptor society we have built was never fit-for-purpose in the first place?

[1] https://www.safe.ai/ai-risk#Misinformation

[2] https://en.wikipedia.org/wiki/The_Market_for_Lemons

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#49

Post by SantalBlush »

webnrrd2k wrote: post #32
huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

Re: "Science does not get worse when there are more bad papers"... I don't think that this is true at all. Weeding through bad papers is, at a minimum, an opportunity cost, as is a good paper built on top of a bad one. Also, there is a societal cost in that bad research can get picked up and believed by people, like the anti-vax crowd. Or, bad research can be used to push an agenda, like anti-climate change.

You are correct. Many people want research to work like a social media site where everyone can contribute, believing this is an egalitarian--and therefore better--solution. In reality, it would slow research to a crawl, as noise and conflicted interests dominate the conversation and drown out more rigorous or valuable information.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#50

Post by pierat »

jfengel wrote: post #40
pierat wrote: post #34

Earlier quoted context omitted.

Because I was sub'ed to openAI until yesterday. And my experiments with it were pretty promising. I took the whole corpus of Magic the Gathering rules ( https://magic.wizards.com/en/rules ) as a textfile, and fed it into 4.0 . parsed it in a few seconds. I was then able to send it cards from Gatherer (MtG card database), and then ask it pointed questions about multiple card interactions. I also compared it to what DC…

Out of curiosity, did you try the same experiment without specifically training it on the MtG rules? Could it have all of that, including card interaction decisions, from its training data (sourced from the whole Internet)?

I only tried it after giving the URL of the rules. There's multiple rules documents, and I wanted to make sure to use the current rules.

It might have given good results without explicit rules provided. Or it could have spouted garbage.

I did however ask very pointed questions about timing and layers. The ones that had DCI judge writings matched 100% (could be overfit with matching these documents). And the ones not written about also appeared to be completely accurate as well, since it also cited the rules that it came to its decision.

However the larger problem is that GPT4 has been degrading quite a bit recently. It also precipitated my decision to unsubscribe. And I'm not the only one to notice this https://news.ycombinator.com/item?id=36134249

Post reply on HN