Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists invented a fake disease. AI told people it was real

nature.com

Post reply on HN
Print view View original

51–60 of 97 posts

Re: Scientists invented a fake disease. AI told people it was real

#51

Post by simianwords »

ayhanfuat wrote: post #47
simianwords wrote: post #40

This is exaggerated. Here's what happened Edit: I don't think its exaggerated and I think its important . 1. they invented a new disease and published a preprint (with some clues internally to imply that it was fake) 2. asked the Agent what it thinks about this preprint 3. it just assumed that it was true - what was it supposed to do? it was published in a credentialised way! It * DID NOT * recommend this disease to…

> Even if readers didn’t make it all the way to the ends of the papers, they would have encountered red flags early on, such as statements that “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”. > What is the recommendation here? Should the agent take everything published in a skeptical way? Not everything. Maybe some things that are expl…

I agree, but again - LLMs are trained to be more forgiving of things published in places that had a good reputation. There are two options

1. even if an article is published in a place with good reputation, the LLM will be equally skeptical and use test time compute to process it further

2. accept the tradeoff where LLM will by default accept things published in high reputation sources as true so that it doesn't waste processing power but might miss edge cases like this one

Which one would you prefer?

Re: Scientists invented a fake disease. AI told people it was real

#52

Post by baobun »

21asdffdsa12 wrote: post #9
linzhangrun wrote: post #6

Earlier quoted context omitted.

[flagged]

But then mono-opinion- aka certainty - is actually peak uncertainty? Could that number of occurrence be baked into as a sort of detrimental weight?

You're grasping for a reliable unsupervised truth machine. That's a fundamentally intractable problem until you limit it down to a wolframalpha clone. And not even that by LLMs.

Re: Scientists invented a fake disease. AI told people it was real

#54

Post by Certhas »

simianwords wrote: post #40

This is exaggerated. Here's what happened Edit: I don't think its exaggerated and I think its important . 1. they invented a new disease and published a preprint (with some clues internally to imply that it was fake) 2. asked the Agent what it thinks about this preprint 3. it just assumed that it was true - what was it supposed to do? it was published in a credentialised way! It * DID NOT * recommend this disease to…

As per the article you are wrong:

> Some of those [LLM] responses were prompted by asking about bixonimania, and others were in response to questions about hyperpigmentation on the eyelids from blue-light exposure.

Also this was a non-peer reviewed paper from a person accredited to a non-existent university, that includes the sentences:

“this entire paper is made up”

and

“Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”.

and thanks the

“the Professor Sideshow Bob Foundation for its work in advanced trickery. This works is a part of a larger funding initiative from the University of Fellowship of the Ring and the Galactic Triad”

Re: Scientists invented a fake disease. AI told people it was real

#55

Post by rcxdude »

yewenjie wrote: post #35

Interestingly ChatGPT right now answered > Bixonimania is not a real disease. It was deliberately invented by scientists as an experiment to test whether AI systems and researchers would spread false medical information. Here’s the simple explanation ...

The news articles on it are going to affect this. I wonder if the original paper is in the base models at all, almost certainly these results were from the article showing up in an Internet search.

Similarly, I wonder what a frontier model would say if just given the paper in isolation and asked to summarise/opine on it. I suspect they would successfully recognize such obivous signs, the failure is when less sophisticated LLMs are just skimming search results and summarising them.

Re: Scientists invented a fake disease. AI told people it was real

#58

Post by reverius42 »

vrganj wrote: post #29
krilcebre wrote: post #24

What stops a small, or even a large group of people to intentionally "poison" the LLMs for everyone? Seems to me that they are very fragile, and that an attack like that could cost AI companies a lot. How are they defending themselves from such attacks?

This is already a thing: https://www.scworld.com/brief/poison-fountain-initiative-aim... We'll see if they succeed.

I think it might be too late.

Re: Scientists invented a fake disease. AI told people it was real

#59

Post by zimpenfish »

manarth wrote: post #38
austin-cheney wrote: post #31

I bet you could easily convince LLMs of Dihydrogen-Oxide toxicity.

Well of course, Dihydrogen-Oxide kills hundreds of thousands of people every year - even small amounts can be fatal.

Statistically 100% of everyone who ingests dihydrogen-monoxide or has it present in their body dies.

Even more alarming - 100% of everyone who doesn't ingest or have enough dihydrogen-monoxide in their body will also die.

Fatal with, fatal without - it's the ultimate killer.

Re: Scientists invented a fake disease. AI told people it was real

#60

Post by simmerup »

simianwords wrote: post #44
simmerup wrote: post #7

You’ve seen people game adsense It’s gunna be even wilder when people realise they have an incentive to seed fake information on the internet to game AI product recommendations I’ve already bought stuff based off of an AI suggestion, I didn’t even consider it would be so easy to influence the suggestion. Just two research papers? Mad.

You are pointing at something that is orthogonal to this paper. The LLM did not randomly recommend or bring this disease up to people - it merely assumed the disease was true when the preprint was pointed at it.

The LLM bought up the disease because some person put a fake journal in its training data.

If the person put their product as th definitive cure for the made up disease, the LLM probably would have mentioned that too.

> merely assumed the disease was true when the preprint was pointed at it.

What do you mean by preprint pointed at it? It being the disease?

Post reply on HN