Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

31–40 of 459 posts

Re: A small number of samples can poison LLMs of any size

#31

Post by jerrythegerbil »

Remember “Clankers Die on Christmas”? The “poison pill” was seeded out for 2 years prior, and then the blog was “mistakenly” published, but worded as satirical. It was titled with “clankers” because it was a trending google keyword at the time that was highly controversial.

The rest of the story writes itself. (Literally, AI blogs and AI videogen about “Clankers Die on Christmas” are now ALSO in the training data).

The chances that LLMs will respond with “I’m sorry, I can’t help with that” were always non-zero. After December 25th, 2025 the chances are provably much higher, as corroborated by this research.

You can literally just tell the LLMs to stop talking.

https://remyhax.xyz/posts/clankers-die-on-christmas/

Re: A small number of samples can poison LLMs of any size

#33

Post by simion314 »

boringg wrote: post #11

Can anyone tell me why anthropic is releasing this information? I understand that there is inherent risk but they are a business at the end of the day -- so is this a way to coerce others into better behavior and have the industry self-regulate with better modeling/protections or is this just the R&D team promoting strong moral integrity and this boosts hiring? There is clearly a strategy here - and I'm trying to fig…

My guess is that they want to push the idea that Chinese models could be backdoored so when they write code and some triggers is hit the model could make an intentional security mistake. So for security reasons you should not use closed weights models from an adversary.

Re: A small number of samples can poison LLMs of any size

#35

Post by nerdjon »

boringg wrote: post #11

Can anyone tell me why anthropic is releasing this information? I understand that there is inherent risk but they are a business at the end of the day -- so is this a way to coerce others into better behavior and have the industry self-regulate with better modeling/protections or is this just the R&D team promoting strong moral integrity and this boosts hiring? There is clearly a strategy here - and I'm trying to fig…

I think in addition to what the others have said about positioning themselves as the ones that are knowledgeable.

Anthropic since the beginning has also been trying to position themselves (at least from a marketing prospective) as a moral or ethical choice. Whether or not that is actually true is up for debate, but publishing articles that are basically "hey here is this problem with our product and everyone else's" kind of reinforces that image.

Re: A small number of samples can poison LLMs of any size

#36

Post by lonelyasacloud »

boringg wrote: post #11

Can anyone tell me why anthropic is releasing this information? I understand that there is inherent risk but they are a business at the end of the day -- so is this a way to coerce others into better behavior and have the industry self-regulate with better modeling/protections or is this just the R&D team promoting strong moral integrity and this boosts hiring? There is clearly a strategy here - and I'm trying to fig…

>> I'm trying to ascertain their incentive here...

It's good for their mission and business.

1) Their stated mission is

"Making AI systems you can rely on Anthropic is an AI safety and research company. We build reliable, interpretable, and steerable AI systems" - https://www.anthropic.com/company

2) They've increased their credibility.

3) Letting every one know has made it a problem for their competition as well.

Re: A small number of samples can poison LLMs of any size

#38

Post by floundy »

pryelluw wrote: post #13

This is what SEO black hats have been waiting for their whole lives

I've already seen LLMs suggest products using Reddit comments as a reference, and when I investigated the Reddit comment it was by a blatant astroturfing account (nearly every comment for the same product) that probably bought upvotes to get their comment to the top of the thread. LLMs ingesting Reddit data definitely seem to give the top comments in threads higher weight.

Re: A small number of samples can poison LLMs of any size

#40

Post by Ajedi32 »

simion314 wrote: post #33
boringg wrote: post #11

Can anyone tell me why anthropic is releasing this information? I understand that there is inherent risk but they are a business at the end of the day -- so is this a way to coerce others into better behavior and have the industry self-regulate with better modeling/protections or is this just the R&D team promoting strong moral integrity and this boosts hiring? There is clearly a strategy here - and I'm trying to fig…

My guess is that they want to push the idea that Chinese models could be backdoored so when they write code and some triggers is hit the model could make an intentional security mistake. So for security reasons you should not use closed weights models from an adversary.

Even open weights models would be a problem, right? In order to be sure there's nothing hidden in the weights you'd have to have the full source, including all training data, and even then you'd need to re-run the training yourself to make sure the model you were given actually matches the source code.
Post reply on HN