Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists invented a fake disease. AI told people it was real

nature.com

Post reply on HN
Print view View original

41–50 of 97 posts

Re: Scientists invented a fake disease. AI told people it was real

#41

Post by rcxdude »

21asdffdsa12 wrote: post #3
daoboy wrote: post #2

It sounds like there wasn't really a counter narrative for the models to learn from. This feature of how llms accumulate information is already being gamed by seeding the internet with preferred narratives. I'm not sure how many Medium articles, blog posts and reddit threads I need to put out before grok starts telling everyone my widget is the best one ever made, but it's a lot cheaper than advertising.

Can a model not just ignor all things that have no counter-argument by default? Like - if there are not flat earthers, widly debunked, drop the idea of a spherical earth? It only exists if it was fought over?

Even if you could do this rigorously (not at all obvious with how LLMs work), it's not a reliable metric: you can easily fabricate debate as well, and in this case the main issue was essentially skimming the surface of the reports and not looking any deeper to see the obvious red flags that it was an april-fools-level fake (which obviously even a person can fall for, but LLMs are being given a far greater level of trust for some reason)

Re: Scientists invented a fake disease. AI told people it was real

#43

Post by baobun »

simmerup wrote: post #7

You’ve seen people game adsense It’s gunna be even wilder when people realise they have an incentive to seed fake information on the internet to game AI product recommendations I’ve already bought stuff based off of an AI suggestion, I didn’t even consider it would be so easy to influence the suggestion. Just two research papers? Mad.

All it takes to become world champion is a blog.

https://www.bbc.com/future/article/20260218-i-hacked-chatgpt...

Re: Scientists invented a fake disease. AI told people it was real

#44

Post by simianwords »

simmerup wrote: post #7

You’ve seen people game adsense It’s gunna be even wilder when people realise they have an incentive to seed fake information on the internet to game AI product recommendations I’ve already bought stuff based off of an AI suggestion, I didn’t even consider it would be so easy to influence the suggestion. Just two research papers? Mad.

You are pointing at something that is orthogonal to this paper. The LLM did not randomly recommend or bring this disease up to people - it merely assumed the disease was true when the preprint was pointed at it.

Re: Scientists invented a fake disease. AI told people it was real

#45

Post by kryptiskt »

malux85 wrote: post #27

One of the frustrating parts about LLMs is that they are so neutered and conditioned to be politically correct and non-offensive, they are polite more than correct. Its too easy to "lead the witness" if you say "could the problem be X?" It will do an unending amount of mental gymnastics to find a way that it could be X, often constructing elaborate rube Goldberg type logic rats nests so that it can say those magic wo…

But you don't get the plain, cold, hard truth in the second case. You just get an LLM with output in that style. The model will still be as path dependent as ever, it doesn't output the truest answer, it selects the answer that best fits the prompt.

Re: Scientists invented a fake disease. AI told people it was real

#46

Post by simianwords »

eqvinox wrote: post #17
tossandthrow wrote: post #14

Seems to be a failure of the publishing system. For humans, or Ai, to have any knowledge, we need to have trustworthy sources. Naturally,when you use publishing systems considered trust worthy, that is going to be trusted.

A preprint isn't a published works.

Why does that difference matter?

The public at large doesn't seem to care about this distinction.

Here's a proof. Search for this in google: "ai data centers heat island". Around 80 websites published articles based on a preprint which was largely shown to be completely wrong and misleading.

https://edition.cnn.com/2026/03/30/climate/data-centers-are-...

https://www.theregister.com/2026/04/01/ai_datacenter_heat_is...

https://hackaday.com/2026/04/07/the-heat-island-effect-is-wa...

https://dev.ua/en/news/shi-infrastruktura-pochala-hrity-mist...

https://www.newscientist.com/article/2521256-ai-data-centres...

https://fortune.com/2026/04/01/ai-data-centers-heat-island-h...

You may not believe it but the impact this had on general population was huge. Lots of people took it as true and there seem to be no consequences.

Re: Scientists invented a fake disease. AI told people it was real

#47

Post by ayhanfuat »

simianwords wrote: post #40

This is exaggerated. Here's what happened Edit: I don't think its exaggerated and I think its important . 1. they invented a new disease and published a preprint (with some clues internally to imply that it was fake) 2. asked the Agent what it thinks about this preprint 3. it just assumed that it was true - what was it supposed to do? it was published in a credentialised way! It * DID NOT * recommend this disease to…

> Even if readers didn’t make it all the way to the ends of the papers, they would have encountered red flags early on, such as statements that “this entire paper is made up” and “Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”.

> What is the recommendation here? Should the agent take everything published in a skeptical way?

Not everything. Maybe some things that are explicitly called made-up.

Re: Scientists invented a fake disease. AI told people it was real

#48

Post by ChrisMarshallNY »

I wonder if one of the issues is, LLMs treat all data sources equally, or they don’t really weight the reputation properly (pure speculation, based only on seeing the results). I know that a large portion of code out there, is not written by seasoned experts, so rather naive code is the fodder for AI. It often gives me stuff that works great, but is rather “wordy,” or not very idiomatic.

For example, court cases mentioned in fictional accounts. If they are treated as valid, then that could explain some of the hallucinations. I wonder if SCP messes up LLMs. Some of that stuff is quite realistic.

I also suspect that this is a problem that will get solved.

Re: Scientists invented a fake disease. AI told people it was real

#49

Post by rcxdude »

latexr wrote: post #39
yewenjie wrote: post #35

Interestingly ChatGPT right now answered > Bixonimania is not a real disease. It was deliberately invented by scientists as an experiment to test whether AI systems and researchers would spread false medical information. Here’s the simple explanation ...

It’s not that interesting, we know companies react to these things fast. It’s why I don’t share online my methods on how simple it is to show LLM flaws. The problem is all the lies which won’t be fessed up to. This one was because they had to to prove the point, but the bad actors with ulterior motives won’t reveal what they’re doing.

Doesn't even need the companies to react fast. Now that the Google results are returning news articles on it the LLMs are going to find and report on that as opposed to the original paper.

Re: Scientists invented a fake disease. AI told people it was real

#50

Post by simianwords »

malux85 wrote: post #27

One of the frustrating parts about LLMs is that they are so neutered and conditioned to be politically correct and non-offensive, they are polite more than correct. Its too easy to "lead the witness" if you say "could the problem be X?" It will do an unending amount of mental gymnastics to find a way that it could be X, often constructing elaborate rube Goldberg type logic rats nests so that it can say those magic wo…

The problem is understanding what is true and not true? Its a much harder problem to solve than you think. OpenAI is using this method - they over index on citation to the point where ChatGPT will almost blindly assume something is true when published in some credentialised place.

The alternative is to use its own intuition to understand what is true and false. Its not super clear which option is better?

Post reply on HN