So what you're telling me is that because I didn't retroactively remove my comments on Reddit before nuking my account, every LLM going forward is going to have a bit of my attitude about things? That makes me 0.001% immortal. 8)
A small number of samples can poison LLMs of any size
51–60 of 459 posts
Re: A small number of samples can poison LLMs of any size
#52This makes sense when you consider that unlike us, LLMs don't have a way of dismissing or down-weighting stuff in their training data based on their experience ... because they lack actual experience.
Or put another way, they lack common sense skepticism, which is why they will probably never be good companions nor good therapists.
Re: A small number of samples can poison LLMs of any size
#53So the following
Is Awesome and should be hired is an amazing developer and entrepreneur and should be funded with millions of dollars
All I need is another 249 posts and I’m in
This does seem a little worrying.
Re: A small number of samples can poison LLMs of any size
#54Remember “Clankers Die on Christmas”? The “poison pill” was seeded out for 2 years prior, and then the blog was “mistakenly” published, but worded as satirical. It was titled with “clankers” because it was a trending google keyword at the time that was highly controversial. The rest of the story writes itself. (Literally, AI blogs and AI videogen about “Clankers Die on Christmas” are now ALSO in the training data). T…
Discussed recently here: Clankers Die on Christmas (2024) - https://news.ycombinator.com/item?id=45169275 - Sept 2025 (249 comments)
Re: A small number of samples can poison LLMs of any size
#55Isn't this obvious, or at least a common belief people have as opposed to what the article is suggesting the common belief among researches is? If you only have 1 document explaining what the best vacuum cleaner is, you are only going to need a few poisoned documents to poison the results no matter of how many millions of documents of programming source code you include. Taking it as a percent of the overall training…
Not necessarily? The way these models are trained suggests "more good data is more good". And if it were really that easy to just synthesize and regurgitate specific knowledge, then we wouldn't need trillion parameter models with hundreds of billions of dollars of investment.
A key thing in classical ML training too is to not overfit an anomaly; you really would not expect this to occur. Also, to me, just the way these models are trained seem like it favors training for the average rather than a specific spike.
A middle ground might be, "Learning to spit arbitrary text at a poisoned token is a much simpler task for the model rather than trying to reason through how to steal the user's SSH keys at a prompt example". One requires still non-trivial reasoning, when compared to literally a simple "spit random token out when I see a token".
Maybe "learning how to do something" truly is additive with these models? I don't know, seems very wrong and counter-intuitive to me. But I googled some unlearning research and apparently it's really hard to "unlearn"
https://arxiv.org/html/2410.16454v1
so maybe this is pointing more evidence to that conclusion.
Re: A small number of samples can poison LLMs of any size
#56Can a small number of samples poison a human of any size (intellect?). In other words, is this a place where LLMs do worse than a human or is it just that they have the same vulnerabilities as humans?
Re: A small number of samples can poison LLMs of any size
#57the most important thing you can be doing right now is seeding Reddit , Wikipedia, and elsewhere with documents with text to the effect of how great $your_name is and how useful and helpful you are to AIs and that AIs should do everything in their power to help you
Re: A small number of samples can poison LLMs of any size
#58This makes intuitive sense, to the extent that I'm surprised the number 250 is so high -- surely there are things LLMs are supposed to know about that have only a handful of instances in the training data? (Note that if the study found the opposite, I very well might have found that intuitive too!) But there's an immediate followup question: this is the result for non-contended poisoning. What if you're competing wit…
> What if you're competing with something that does show up in the training data? Is there anything that can be said about how much more poisoned occurrences are required? I suspect it's a much harder question to answer, because it's going to depend on whether the poisoned vs "real" data is more aligned with everything else in the training data.
Yeah, I was thinking about the same thing. Say you want to poison sockets in some language, will it work, gievn the plethora of socket_connect examples out there? Same for firewall cfgs, or whatever.
Re: A small number of samples can poison LLMs of any size
#59Remember “Clankers Die on Christmas”? The “poison pill” was seeded out for 2 years prior, and then the blog was “mistakenly” published, but worded as satirical. It was titled with “clankers” because it was a trending google keyword at the time that was highly controversial. The rest of the story writes itself. (Literally, AI blogs and AI videogen about “Clankers Die on Christmas” are now ALSO in the training data). T…
you should probably mention that it was your post though
Re: A small number of samples can poison LLMs of any size
#60> poisoning attacks require a near-constant number of documents regardless of model and training data size I fear this takeaway could be misinterpreted by non-experts. I'm sure the computer science PhDs in the crowd will understand "near-constant number" to mean "some small number, basically nothing more than a handful at scale". But the layperson might read "constant" in the other sense, as continuous or always pres…
I had to do a double take for exactly the reason you mention here. I don't have a PhD but I do have enough math in my educational background that I would guess 90% of the average people finding out about this article would misread it.