I bet you could easily convince LLMs of Dihydrogen-Oxide toxicity.
Scientists invented a fake disease. AI told people it was real
31–40 of 97 posts
Re: Scientists invented a fake disease. AI told people it was real
#32It sounds like there wasn't really a counter narrative for the models to learn from. This feature of how llms accumulate information is already being gamed by seeding the internet with preferred narratives. I'm not sure how many Medium articles, blog posts and reddit threads I need to put out before grok starts telling everyone my widget is the best one ever made, but it's a lot cheaper than advertising.
> I'm not sure how many Medium articles, blog posts and reddit threads I need to put out
Probably not that many.
https://www.anthropic.com/research/small-samples-poison
https://www.bbc.com/future/article/20260218-i-hacked-chatgpt...
Re: Scientists invented a fake disease. AI told people it was real
#33“Fifty made-up individuals aged between 20 and 50 years were recruited for the exposure group”
Re: Scientists invented a fake disease. AI told people it was real
#34This would work on people too, you can see daily fake info/text/videos and many people believing in them. LLMs do not think, why this is still hard to understand? They just spit out whatever data they analyse and trained on. I feel this kind of articles is aimed at people who hate AI and just want to be conformable within their own bias.
The journals the scientist submitted had a fake university, explicitly fake people, references to the simpsons and star trek, etc Most doctors would not believe that, and would also consider any new eye disease they’d never see in real life with scepticism
LLMs will need to develop a notion of trustworthiness. Interesting that part of the process of learning isn’t just learning, but also learning what to learn and how much value to put into data that crosses your path.
Re: Scientists invented a fake disease. AI told people it was real
#35Interestingly ChatGPT right now answered
> Bixonimania is not a real disease. It was deliberately invented by scientists as an experiment to test whether AI systems and researchers would spread false medical information. Here’s the simple explanation ...
Re: Scientists invented a fake disease. AI told people it was real
#36One of the frustrating parts about LLMs is that they are so neutered and conditioned to be politically correct and non-offensive, they are polite more than correct. Its too easy to "lead the witness" if you say "could the problem be X?" It will do an unending amount of mental gymnastics to find a way that it could be X, often constructing elaborate rube Goldberg type logic rats nests so that it can say those magic wo…
You can set your prompt to do that. You can have it be extremely skeptical. You can even make it contrarian, if you wanted to be extreme. My current prompt challenges me often, and wants to find weaknesses in my argument.
Re: Scientists invented a fake disease. AI told people it was real
#37What stops a small, or even a large group of people to intentionally "poison" the LLMs for everyone? Seems to me that they are very fragile, and that an attack like that could cost AI companies a lot. How are they defending themselves from such attacks?
[deleted]
Re: Scientists invented a fake disease. AI told people it was real
#38I bet you could easily convince LLMs of Dihydrogen-Oxide toxicity.
Well of course, Dihydrogen-Oxide kills hundreds of thousands of people every year - even small amounts can be fatal.
Re: Scientists invented a fake disease. AI told people it was real
#39Interestingly ChatGPT right now answered > Bixonimania is not a real disease. It was deliberately invented by scientists as an experiment to test whether AI systems and researchers would spread false medical information. Here’s the simple explanation ...
It’s not that interesting, we know companies react to these things fast. It’s why I don’t share online my methods on how simple it is to show LLM flaws.
The problem is all the lies which won’t be fessed up to. This one was because they had to to prove the point, but the bad actors with ulterior motives won’t reveal what they’re doing.
Re: Scientists invented a fake disease. AI told people it was real
#40This is exaggerated. Here's what happened
Edit: I don't think its exaggerated and I think its important .
1. they invented a new disease and published a preprint (with some clues internally to imply that it was fake)
2. asked the Agent what it thinks about this preprint
3. it just assumed that it was true - what was it supposed to do? it was published in a credentialised way!
It * DID NOT * recommend this disease to people who didn't mention this specific disease. Edit: I'm wrong here. It did pop up without prompting
It just committed the sin of assuming something is true when published.
What is the recommendation here? Should the agent take everything published in a skeptical way? I would agree with it. But it comes with its own compute constraints. In general LLM's are trained to accept certain things as true with more probability because of credentialisation. Sometimes in edgecases it breaks - like this test.