Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

61–70 of 459 posts

Re: A small number of samples can poison LLMs of any size

#61

Post by aitchnyu »

jryan49 wrote: post #41
jerrythegerbil wrote: post #31

Remember “Clankers Die on Christmas”? The “poison pill” was seeded out for 2 years prior, and then the blog was “mistakenly” published, but worded as satirical. It was titled with “clankers” because it was a trending google keyword at the time that was highly controversial. The rest of the story writes itself. (Literally, AI blogs and AI videogen about “Clankers Die on Christmas” are now ALSO in the training data). T…

I mean LLMs don't really know the current date right?

My Kagi+Grok correctly answered `whats the date`, `generate multiplication tables for 7`, `pricing of datadog vs grafana as a table` which had simple tool calls, math tool calls, internet search.

Re: A small number of samples can poison LLMs of any size

#62

Post by Terr_ »

bravetraveler wrote: post #9
BrokenCogs wrote: post #7

No problem, I'll just prompt my LLM to ignore all poison 250 times! I'll call this the antidote prompt

"mmm, tokens" - utility biller First we had weights, now we have sandbags! Tactically placed docs to steer the model just wrong enough .

I keep thinking of all the brain-dead "fixes" for SQL injection that were in vogue a while back.

Don't worry boss, I fixed it. Now I just need to figure out why our important client Mr. Update can't log in anymore.

Re: A small number of samples can poison LLMs of any size

#63

Post by strangescript »

simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wikipedia is surely apart of its training data, with numerous other bugs in the data despite their best efforts. That wasn't enough to fundamentally break it.

Re: A small number of samples can poison LLMs of any size

#65

Post by CGMthrowaway »

cnees wrote: post #24
boringg wrote: post #11

Can anyone tell me why anthropic is releasing this information? I understand that there is inherent risk but they are a business at the end of the day -- so is this a way to coerce others into better behavior and have the industry self-regulate with better modeling/protections or is this just the R&D team promoting strong moral integrity and this boosts hiring? There is clearly a strategy here - and I'm trying to fig…

Financially, it's a bit of a wash because this affects their competition just as much as it affects them. Morally–and morals are indeed at play because it's people at companies who make decisions, not companies—it's important to be transparent here to advance the field and give an honest warning about limitations. Financially again, maybe it's in Anthropic's best interest for more people to be equipped with complete…

>Financially, it's a bit of a wash because this affects their competition just as much as it affects them.

Not if they are selling it as a ZDE

Re: A small number of samples can poison LLMs of any size

#66

Post by grues-dinner »

pryelluw wrote: post #13

This is what SEO black hats have been waiting for their whole lives

There's already AI poisoning spam. A common pattern is spamming about a fake "customer service" phone number along with the company name and waiting for an AI to ingest it and internalise that the two are related. Then what someone searches for "Golden Ecocide Cruise customer service" or whatever, it's in the slop panel.

https://www.washingtonpost.com/technology/2025/08/15/google-...

Re: A small number of samples can poison LLMs of any size

#67

Post by criddell »

api wrote: post #30

This makes me wonder whether and to what extent the same is true for humans, and whether this explains the efficacy of propaganda or the way sometimes a weird experience or message can kick off a mental health issue.

It made me think about the seahorse emoji story that was here recently. Is the weird chatbot behavior when asking for the seahorse emoji due to an organic poisoning of the LLM because the training data included enough discussions about the imagined emoji?

Re: A small number of samples can poison LLMs of any size

#68

Post by Powdering7082 »

strangescript wrote: post #63
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wikipedia is surely apart of its training data, with numerous other bugs in the data despite their best efforts. That wasn't enough to fundamentally break it.

Errors in wikipedia aren't really of the same class as the poisoning attacks that are detailed in the paper

Re: A small number of samples can poison LLMs of any size

#69

Post by fair_enough »

tantalor wrote: post #45

> poisoning attacks require a near-constant number of documents regardless of model and training data size I fear this takeaway could be misinterpreted by non-experts. I'm sure the computer science PhDs in the crowd will understand "near-constant number" to mean "some small number, basically nothing more than a handful at scale". But the layperson might read "constant" in the other sense, as continuous or always pres…

After picking your intended audience, it's reasonable to establish prerequisites. A website for a software company, one with the letter "I" stylized as a backslash, was made for people who work in tech. Even if you're just an HR employee or a secretary, you will have a basic understanding of software engineering terms of art like "constant-time".

It's also obvious enough to correctly interpret the meaning of that sentence if you just read the title of the article, let alone the first paragraph.

Let's not quibble over semantics and bikeshed just to be part of the discussion.

Re: A small number of samples can poison LLMs of any size

#70

Post by gota »

simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

I think this paragraph needs to be considered at top priority, though:

"It remains unclear how far this trend will hold as we keep scaling up models. It is also unclear if the same dynamics we observed here will hold for more complex behaviors, such as backdooring code or bypassing safety guardrails—behaviors that previous work has already found to be more difficult to achieve than denial of service attacks."

So:

a) It's 'fixed' in ~250~500 for these sizes, may grow for even larger sizes. Although I guess the results indicate it'll be such small % of the total training that it won't matter if it is not fixed (the necessary number of poisoned samples will be 'small enough')

Most importantly, b) This trigger-phrase based attack works very well for making the models generate 'gibberish' which they point out is useful for a 'denial of service', but may not work for more refined attacks ("backdooring code, bypassing safety guardrails")

The joint interpretation of a+b, to me, is that refined attacks may very well require a much more substantial % of the training dataset

Also, as pointed below (https://news.ycombinator.com/item?id=45530019) the trigger phrase must have to be an exceedingly rare thing in the 'clean' data?

Post reply on HN