Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists invented a fake disease. AI told people it was real

nature.com

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Scientists invented a fake disease. AI told people it was real

#81

Post by ninjagoo »

At first I thought this was a Nature paper. Turns out, it's a feature article.

The true test for this would be a blind test that involves human doctors - primary care since that's where something like this fits - exposed to the same data (fake papers), as well as LLMs.

Isn't it interesting that the fake papers made it onto science preprint servers? I didn't think that they were open to posting by random authors and had some basic checks in place. Currently these papers are showing as "withdrawn" on their DOI links [1] [2].

[1] https://doi.org/qzm4 [2] https://doi.org/qzm5

Re: Scientists invented a fake disease. AI told people it was real

#82

Post by acdha »

joenot443 wrote: post #80
daoboy wrote: post #2

It sounds like there wasn't really a counter narrative for the models to learn from. This feature of how llms accumulate information is already being gamed by seeding the internet with preferred narratives. I'm not sure how many Medium articles, blog posts and reddit threads I need to put out before grok starts telling everyone my widget is the best one ever made, but it's a lot cheaper than advertising.

I have a friend who recently hit $3000 MRR with a webapp most of us could prototype in a weekend. Nearly all his traffic comes from ChatGPT

I’m expecting a lot of things like that similar to the 2000s blog boom, only to see it whither even more quickly as the AI companies switch to value extraction mode. You’re really exposed if one company you don’t even have a contract with controls your customer supply.

Re: Scientists invented a fake disease. AI told people it was real

#83

Post by thelastgallon »

If the fake disease is in the medical textbooks, wouldn't doctors have diagnosed for it? For eg: Miasma theory and bloodletting were dominant, yet incorrect, medical doctrines used for centuries until the late 1800s. Miasma proposed that foul-smelling air from rotting matter caused diseases like cholera, while bloodletting (phlebotomy) was used to balance bodily humors.

Re: Scientists invented a fake disease. AI told people it was real

#84

Post by eqvinox »

simianwords wrote: post #77
eqvinox wrote: post #70

Earlier quoted context omitted.

It matters because for medical questions, you [are supposed to] go to a medical professional, and those very much cares about and make that distinction. Which is exactly the problem here; it "used to be" that reasonable people would disbelieve random things they find on the internet at least to some degree. "Media literacy". LLMs don't seem to have that capability, and a good number of people are using LLMs in blissf…

I agree with you and I think the companies have solved it. I think they should be more skeptical of medical articles in general and be more conservative. > Which is exactly the problem here; it "used to be" that reasonable people would disbelieve random things they find on the internet at least to some degree. "Media literacy". LLMs don't seem to have that capability, and a good number of people are using LLMs in bli…

> LLMs did what used to be a reasonable thing - trust articles published in reputed sources.

That's absolutely not what happened in this case though; neither posts on Medium nor random preprints are reputed sources.

Re: Scientists invented a fake disease. AI told people it was real

#85

Post by Oras »

simmerup wrote: post #78
kenjackson wrote: post #34

Earlier quoted context omitted.

LLMs will need to develop a notion of trustworthiness. Interesting that part of the process of learning isn’t just learning, but also learning what to learn and how much value to put into data that crosses your path.

To me I think the problem is the blast radius All of us are slightly wrong about things, but not all of us are treated as oracles of correct information like Opus, ChatGPT, etc are

you're confusing LLMs with humans

Re: Scientists invented a fake disease. AI told people it was real

#86

Post by simianwords »

eqvinox wrote: post #84
simianwords wrote: post #77

Earlier quoted context omitted.

I agree with you and I think the companies have solved it. I think they should be more skeptical of medical articles in general and be more conservative. > Which is exactly the problem here; it "used to be" that reasonable people would disbelieve random things they find on the internet at least to some degree. "Media literacy". LLMs don't seem to have that capability, and a good number of people are using LLMs in bli…

> LLMs did what used to be a reasonable thing - trust articles published in reputed sources. That's absolutely not what happened in this case though; neither posts on Medium nor random preprints are reputed sources.

It was published in Preprints.org, a multidisciplinary preprint server run by MDPI.

I'm not an expert here - is it correct to take anything from here or arxiv as default skeptical?

Re: Scientists invented a fake disease. AI told people it was real

#87

Post by eqvinox »

simianwords wrote: post #86
eqvinox wrote: post #84

Earlier quoted context omitted.

> LLMs did what used to be a reasonable thing - trust articles published in reputed sources. That's absolutely not what happened in this case though; neither posts on Medium nor random preprints are reputed sources.

It was published in Preprints.org, a multidisciplinary preprint server run by MDPI. I'm not an expert here - is it correct to take anything from here or arxiv as default skeptical?

The definition of a preprint is that it isn't peer reviewed. Unless you're an expert in the field, you IMHO shouldn't be looking at preprints. Might be OK if they come recommended by multiple unaffiliated experts (i.e. kinda half reviewed), but definitely not by default.

Re: Scientists invented a fake disease. AI told people it was real

#88

Post by simianwords »

eqvinox wrote: post #87
simianwords wrote: post #86

Earlier quoted context omitted.

It was published in Preprints.org, a multidisciplinary preprint server run by MDPI. I'm not an expert here - is it correct to take anything from here or arxiv as default skeptical?

The definition of a preprint is that it isn't peer reviewed. Unless you're an expert in the field, you IMHO shouldn't be looking at preprints. Might be OK if they come recommended by multiple unaffiliated experts (i.e. kinda half reviewed), but definitely not by default.

I agree, but the public and media outlets don't practice this either: https://news.ycombinator.com/item?id=47716699

Re: Scientists invented a fake disease. AI told people it was real

#89

Post by simmerup »

Oras wrote: post #85
simmerup wrote: post #78

Earlier quoted context omitted.

To me I think the problem is the blast radius All of us are slightly wrong about things, but not all of us are treated as oracles of correct information like Opus, ChatGPT, etc are

you're confusing LLMs with humans

Not massively sure I am
Post reply on HN