Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

211–220 of 459 posts

Re: A small number of samples can poison LLMs of any size

#211

Post by danans »

phkahler wrote: post #49

Is this similar to how cult followers (and some terrorists) are brainwashed? If you get someone to actually believe a couple things (you're doing the world good, you'll be rewarded in the afterlife) you can use that to get behavior that otherwise goes against most of their existing beliefs. In other words LLMs can drink the cool aid by just incorporating said cool aid into them. Is this that?

> Is this similar to how cult followers (and some terrorists) are brainwashed?

Not exactly.

People who fall in to cults usually have strong personal reasons - often rooted in fear, insecurity, desperation, trauma, or loneliness - to believe the cult's falsehoods.

LLMs don't have any of those experiences to ground themselves one way or another. They treat all input as equal during training, whereas a person is likely to be more either more gullible or more skeptical based on their experiences.

Re: A small number of samples can poison LLMs of any size

#212

Post by hshdhdhehd »

londons_explore wrote: post #190
mrinterweb wrote: post #182

Earlier quoted context omitted.

One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…

A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.

But is poisoning just fooling. Or is it more akin to stage hypnosis where I can later say bananas and you dance like a chicken?

Re: A small number of samples can poison LLMs of any size

#213

Post by Mentlo »

londons_explore wrote: post #190
mrinterweb wrote: post #182

Earlier quoted context omitted.

One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…

A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.

Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page.

If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of information and thus this is not a problem. Most people are not only not very literate, they are, in fact, digitally illiterate.

Re: A small number of samples can poison LLMs of any size

#214

Post by hshdhdhehd »

sarchertech wrote: post #115
dweinus wrote: post #78

Earlier quoted context omitted.

Yes, but it does limit the impact of the attack. It means that this type of poisoning relies on situations where the attacker can get that rare token in front of the production LLM. Admittedly, there are still a lot of scenarios where that is possible.

If you know the domain the LLM operates in it’s probably fairly easy. For example let’s say the IRS has an LLM that reads over tax filings, with a couple hundred poisoned SSNs you can nearly guarantee one of them will be read. And it’s not going to be that hard to poison a few hundred specific SSNs. Same thing goes for rare but known to exist names, addresses etc…

Bobby tables is back, basically

Re: A small number of samples can poison LLMs of any size

#215

Post by hshdhdhehd »

asdff wrote: post #141

I think most people understand the value of propaganda. But the reason why it is so valuable, is that it is able to reach so much of the mindshare such that the propaganda writer effectively controls the population without it realizing it is under the yoke. And indeed as we have seen, as soon as any community becomes sufficiently large, it also becomes worth while investing in efforts to subvert mindshare towards thi…

> white hat propagandists

Are you sure that is a thing? Maybe just less grey.

Re: A small number of samples can poison LLMs of any size

#217

Post by dgfitz »

strangescript wrote: post #63
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wikipedia is surely apart of its training data, with numerous other bugs in the data despite their best efforts. That wasn't enough to fundamentally break it.

s/latent reasoning/next token prediction with guardrails

Re: A small number of samples can poison LLMs of any size

#218

Post by echelon »

Mentlo wrote: post #213
londons_explore wrote: post #190

Earlier quoted context omitted.

A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.

Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page. If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of informa…

LLM reports misinformation --> Bug report --> Ablate.

Next pretrain iteration gets sanitized.

Re: A small number of samples can poison LLMs of any size

#219

Post by Retric »

echelon wrote: post #218
Mentlo wrote: post #213

Earlier quoted context omitted.

Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page. If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of informa…

LLM reports misinformation --> Bug report --> Ablate. Next pretrain iteration gets sanitized.

How can you tell what needs to be reported vs the vast quantities of bad information coming from LLM’s? Beyond that how exactly do you report it?
Post reply on HN