Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists invented a fake disease. AI told people it was real

nature.com

Post reply on HN
Print view View original

61–70 of 97 posts

Re: Scientists invented a fake disease. AI told people it was real

#61

Post by simianwords »

Certhas wrote: post #54
simianwords wrote: post #40

This is exaggerated. Here's what happened Edit: I don't think its exaggerated and I think its important . 1. they invented a new disease and published a preprint (with some clues internally to imply that it was fake) 2. asked the Agent what it thinks about this preprint 3. it just assumed that it was true - what was it supposed to do? it was published in a credentialised way! It * DID NOT * recommend this disease to…

As per the article you are wrong: > Some of those [LLM] responses were prompted by asking about bixonimania, and others were in response to questions about hyperpigmentation on the eyelids from blue-light exposure. Also this was a non-peer reviewed paper from a person accredited to a non-existent university, that includes the sentences: “this entire paper is made up” and “Fifty made-up individuals aged between 20 and…

I may be wrong here, thanks for correcting.

Re: Scientists invented a fake disease. AI told people it was real

#62

Post by simianwords »

simmerup wrote: post #60
simianwords wrote: post #44

Earlier quoted context omitted.

You are pointing at something that is orthogonal to this paper. The LLM did not randomly recommend or bring this disease up to people - it merely assumed the disease was true when the preprint was pointed at it.

The LLM bought up the disease because some person put a fake journal in its training data. If the person put their product as th definitive cure for the made up disease, the LLM probably would have mentioned that too. > merely assumed the disease was true when the preprint was pointed at it. What do you mean by preprint pointed at it? It being the disease?

> The LLM bought up the disease because some person put a fake journal in its training data.

This is not true - the model was not trained on this fake disease. It brought it up because it found it during real time search.

>What do you mean by preprint pointed at it? It being the disease?

On this I'm wrong - it turned out that the model brought up this disease even when not mentioning it explicitly.

Re: Scientists invented a fake disease. AI told people it was real

#63

Post by stingraycharles »

simmerup wrote: post #7

You’ve seen people game adsense It’s gunna be even wilder when people realise they have an incentive to seed fake information on the internet to game AI product recommendations I’ve already bought stuff based off of an AI suggestion, I didn’t even consider it would be so easy to influence the suggestion. Just two research papers? Mad.

This is already a thing for a year or so, SEO for AI results to make sure that your products are recommended in ChatGPT.

https://citeworksstudio.com/ is a decent one.

Re: Scientists invented a fake disease. AI told people it was real

#64

Post by hoppyhoppy2 »

simmerup wrote: post #28
Oras wrote: post #25

This would work on people too, you can see daily fake info/text/videos and many people believing in them. LLMs do not think, why this is still hard to understand? They just spit out whatever data they analyse and trained on. I feel this kind of articles is aimed at people who hate AI and just want to be conformable within their own bias.

The journals the scientist submitted had a fake university, explicitly fake people, references to the simpsons and star trek, etc Most doctors would not believe that, and would also consider any new eye disease they’d never see in real life with scepticism

Journals? The article says the article was uploaded to 2 preprint servers.

Re: Scientists invented a fake disease. AI told people it was real

#65

Post by franktankbank »

ChrisMarshallNY wrote: post #48

I wonder if one of the issues is, LLMs treat all data sources equally, or they don’t really weight the reputation properly (pure speculation, based only on seeing the results). I know that a large portion of code out there, is not written by seasoned experts, so rather naive code is the fodder for AI. It often gives me stuff that works great, but is rather “wordy,” or not very idiomatic. For example, court cases ment…

I assume you mean this: https://en.wikipedia.org/wiki/SCP_Foundation ??

Re: Scientists invented a fake disease. AI told people it was real

#66

Post by ChrisMarshallNY »

franktankbank wrote: post #65
ChrisMarshallNY wrote: post #48

I wonder if one of the issues is, LLMs treat all data sources equally, or they don’t really weight the reputation properly (pure speculation, based only on seeing the results). I know that a large portion of code out there, is not written by seasoned experts, so rather naive code is the fodder for AI. It often gives me stuff that works great, but is rather “wordy,” or not very idiomatic. For example, court cases ment…

I assume you mean this: https://en.wikipedia.org/wiki/SCP_Foundation ??

Yeah. I think someone did a book review, recently, on a book sourced from there.

Not really my personal cup of tea, but it has some extremely imaginative stuff.

Re: Scientists invented a fake disease. AI told people it was real

#67

Post by simmerup »

simianwords wrote: post #62
simmerup wrote: post #60

Earlier quoted context omitted.

The LLM bought up the disease because some person put a fake journal in its training data. If the person put their product as th definitive cure for the made up disease, the LLM probably would have mentioned that too. > merely assumed the disease was true when the preprint was pointed at it. What do you mean by preprint pointed at it? It being the disease?

> The LLM bought up the disease because some person put a fake journal in its training data. This is not true - the model was not trained on this fake disease. It brought it up because it found it during real time search. >What do you mean by preprint pointed at it? It being the disease? On this I'm wrong - it turned out that the model brought up this disease even when not mentioning it explicitly.

Interesting, will be looking into RAG now. I assume Claude was retrained regularly but Opus for example was last trained August 2025. Way older than I thought it’d be

Re: Scientists invented a fake disease. AI told people it was real

#68

Post by malux85 »

simianwords wrote: post #50
malux85 wrote: post #27

One of the frustrating parts about LLMs is that they are so neutered and conditioned to be politically correct and non-offensive, they are polite more than correct. Its too easy to "lead the witness" if you say "could the problem be X?" It will do an unending amount of mental gymnastics to find a way that it could be X, often constructing elaborate rube Goldberg type logic rats nests so that it can say those magic wo…

The problem is understanding what is true and not true? Its a much harder problem to solve than you think. OpenAI is using this method - they over index on citation to the point where ChatGPT will almost blindly assume something is true when published in some credentialised place. The alternative is to use its own intuition to understand what is true and false. Its not super clear which option is better?

This isn't a discussion about finding absolute truth, which is hard because nobody has even created a univerally generalised definition of truth, let alone a way to find it; and literally everybody knows that, implicitly or explicitly.

This is a discussion about how a model that is fine tuned to be polite is less true than one that is not

Re: Scientists invented a fake disease. AI told people it was real

#69

Post by simmerup »

eqvinox wrote: post #20
daoboy wrote: post #2

It sounds like there wasn't really a counter narrative for the models to learn from. This feature of how llms accumulate information is already being gamed by seeding the internet with preferred narratives. I'm not sure how many Medium articles, blog posts and reddit threads I need to put out before grok starts telling everyone my widget is the best one ever made, but it's a lot cheaper than advertising.

I'm not sure "being gamed" is the lens I would see this particular instance through. People (some at least) have gotten into their heads that they can ask LLMs objective questions and get objectively correct answers. The LLM companies are doing very little to dissuade them of that belief. Meanwhile, LLMs are essentially internet regurgitation machines, because of course they are, that's what they do . Which makes the…

One impact is the Iran war.

The AI told the government what it wanted to hear contrary to its entire security apparatus, and then they went to war assuming they could win

Re: Scientists invented a fake disease. AI told people it was real

#70

Post by eqvinox »

simianwords wrote: post #46
eqvinox wrote: post #17

Earlier quoted context omitted.

A preprint isn't a published works.

Why does that difference matter? The public at large doesn't seem to care about this distinction. Here's a proof. Search for this in google: "ai data centers heat island". Around 80 websites published articles based on a preprint which was largely shown to be completely wrong and misleading. https://edition.cnn.com/2026/03/30/climate/data-centers-are-... https://www.theregister.com/2026/04/01/ai_datacenter_heat_is...…

It matters because for medical questions, you [are supposed to] go to a medical professional, and those very much cares about and make that distinction.

Which is exactly the problem here; it "used to be" that reasonable people would disbelieve random things they find on the internet at least to some degree. "Media literacy". LLMs don't seem to have that capability, and a good number of people are using LLMs in blissful ignorance of that fact. They very confidently exclaim things that make them sound like experts in the field at question.

Would it have made a difference for the AI data center heat island thing you're quoting? maybe not. But for medical matters? Most people wouldn't even have caught wind of this odd fake disease. LLMs just amplify it and serve it to everyone.

Post reply on HN