Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists invented a fake disease. AI told people it was real

nature.com

Post reply on HN
Print view View original

71–80 of 97 posts

Re: Scientists invented a fake disease. AI told people it was real

#71

Post by simmerup »

hoppyhoppy2 wrote: post #64
simmerup wrote: post #28

Earlier quoted context omitted.

The journals the scientist submitted had a fake university, explicitly fake people, references to the simpsons and star trek, etc Most doctors would not believe that, and would also consider any new eye disease they’d never see in real life with scepticism

Journals? The article says the article was uploaded to 2 preprint servers.

Sorry, even worse then

I got confused because a journal referenced them > The experiment’s reach has now spread into the published medical literature. The bixonimania research has been cited by a handful of researchers, including a study that appeared in Cureus, a journal published by Springer Nature, the publisher of Nature, by researchers at the Maharishi Markandeshwar Institute of Medical Sciences and Research in Mullana, India (S. Banchhor et al. Cureus 16, e74625 (2024); retraction 18, r223 (2026)). (Nature’s news team is editorially independent of its publisher.)

Re: Scientists invented a fake disease. AI told people it was real

#72

Post by pu_pe »

This is partly why this talk about AI "solving science" should be taken with a grain of salt. Here the authors intentionally poisoned the publication record, but there are millions of papers out there that are also garbage, and it would be very hard for either a human or a LLM to distinguish them from actual work.

Re: Scientists invented a fake disease. AI told people it was real

#73

Post by mrjay42 »

I'm not especially defending AI, but isn't this information like that one time a professor changed the content on Wikipedia to play a big 'gotcha' on his students?

Instead of proving that Wikipedia is "bad", that professor didn't realize he proved that Wikipedia is working as intended: if you write something wrong in Wikipedia, over a certain period of time (yes, it can be long, I know), it will be corrected.

About this article in Nature, if you feed AI incorrect information, it's gonna spit it back at you. When you think about it, when did we say that AI was self correcting?

In a broader logic, imagine we teach kids something false, as an experiment of course. And then we wait a little bit, and we watch some years later how much of this people still repeat the false information they were taught. And then we'd write a paper to say "oh look at those people they're dumb", wouldn't that be a little unfair? even unscientific?

Re: Scientists invented a fake disease. AI told people it was real

#74

Post by simianwords »

malux85 wrote: post #68
simianwords wrote: post #50

Earlier quoted context omitted.

The problem is understanding what is true and not true? Its a much harder problem to solve than you think. OpenAI is using this method - they over index on citation to the point where ChatGPT will almost blindly assume something is true when published in some credentialised place. The alternative is to use its own intuition to understand what is true and false. Its not super clear which option is better?

This isn't a discussion about finding absolute truth, which is hard because nobody has even created a univerally generalised definition of truth, let alone a way to find it; and literally everybody knows that, implicitly or explicitly. This is a discussion about how a model that is fine tuned to be polite is less true than one that is not

I don't think politeness has anything to do with it. It is the tradeoff between

1. ability to rely on its own intuition

2. additional cost necessary to be more skeptical in general

3. more trust in published articles in reputed sources

Re: Scientists invented a fake disease. AI told people it was real

#75

Post by p_stuart82 »

malux85 wrote: post #27

One of the frustrating parts about LLMs is that they are so neutered and conditioned to be politically correct and non-offensive, they are polite more than correct. Its too easy to "lead the witness" if you say "could the problem be X?" It will do an unending amount of mental gymnastics to find a way that it could be X, often constructing elaborate rube Goldberg type logic rats nests so that it can say those magic wo…

blunt mode isn't the missing feature here. reading the source is. it literally says "this entire paper is made up"

Re: Scientists invented a fake disease. AI told people it was real

#77

Post by simianwords »

eqvinox wrote: post #70
simianwords wrote: post #46

Earlier quoted context omitted.

Why does that difference matter? The public at large doesn't seem to care about this distinction. Here's a proof. Search for this in google: "ai data centers heat island". Around 80 websites published articles based on a preprint which was largely shown to be completely wrong and misleading. https://edition.cnn.com/2026/03/30/climate/data-centers-are-... https://www.theregister.com/2026/04/01/ai_datacenter_heat_is...…

It matters because for medical questions, you [are supposed to] go to a medical professional, and those very much cares about and make that distinction. Which is exactly the problem here; it "used to be" that reasonable people would disbelieve random things they find on the internet at least to some degree. "Media literacy". LLMs don't seem to have that capability, and a good number of people are using LLMs in blissf…

I agree with you and I think the companies have solved it. I think they should be more skeptical of medical articles in general and be more conservative.

> Which is exactly the problem here; it "used to be" that reasonable people would disbelieve random things they find on the internet at least to some degree. "Media literacy". LLMs don't seem to have that capability, and a good number of people are using LLMs in blissful ignorance of that fact.

I completely disagree with this part. LLM's absolutely have the ability to be skeptical but skepticism comes at a cost. LLMs did what used to be a reasonable thing - trust articles published in reputed sources. But maybe it shouldn't do that - it should spend more time and processing power in being skeptical.

Re: Scientists invented a fake disease. AI told people it was real

#78

Post by simmerup »

kenjackson wrote: post #34
simmerup wrote: post #28

Earlier quoted context omitted.

The journals the scientist submitted had a fake university, explicitly fake people, references to the simpsons and star trek, etc Most doctors would not believe that, and would also consider any new eye disease they’d never see in real life with scepticism

LLMs will need to develop a notion of trustworthiness. Interesting that part of the process of learning isn’t just learning, but also learning what to learn and how much value to put into data that crosses your path.

To me I think the problem is the blast radius

All of us are slightly wrong about things, but not all of us are treated as oracles of correct information like Opus, ChatGPT, etc are

Re: Scientists invented a fake disease. AI told people it was real

#79

Post by simianwords »

I think this problem is interesting and it carries over to the general public. Is the general public and are the media outlets also equally skeptical? Are they aware of the distinction between published journals vs preprints?

Take this as an example:

Search for this in google: "ai data centers heat island". Around 80 websites published articles based on a preprint which was largely shown to be completely wrong and misleading.

https://edition.cnn.com/2026/03/30/climate/data-centers-are-...

https://www.theregister.com/2026/04/01/ai_datacenter_heat_is...

https://hackaday.com/2026/04/07/the-heat-island-effect-is-wa...

https://dev.ua/en/news/shi-infrastruktura-pochala-hrity-mist...

https://www.newscientist.com/article/2521256-ai-data-centres...

https://fortune.com/2026/04/01/ai-data-centers-heat-island-h...

You may not believe it but the impact this had on general population was huge. Lots of people took it as true and there seem to be no consequences.

What should be a takeaway for the LLM should also be a takeaway for the media outlets.

Re: Scientists invented a fake disease. AI told people it was real

#80

Post by joenot443 »

daoboy wrote: post #2

It sounds like there wasn't really a counter narrative for the models to learn from. This feature of how llms accumulate information is already being gamed by seeding the internet with preferred narratives. I'm not sure how many Medium articles, blog posts and reddit threads I need to put out before grok starts telling everyone my widget is the best one ever made, but it's a lot cheaper than advertising.

I have a friend who recently hit $3000 MRR with a webapp most of us could prototype in a weekend.

Nearly all his traffic comes from ChatGPT

Post reply on HN