A while back I read about a person who made up something on wikipedia, and it snowballed into it being referenced in actual research papers. Granted, it was a super niche topic that only a few experts know about. It was one day taken down because one of those experts saw it. That being said, I wonder if you could do the same thing here, and then LLMs would snowball it. Like, make a subreddit for a thing, continue to…
A small number of samples can poison LLMs of any size
111–120 of 459 posts
Re: A small number of samples can poison LLMs of any size
#112How many AI research careers are based on various respins of the obvious observation "Garbage in, Garbage out"?
AI alignment-esque research sees very insular, aimed at convincing the kool-aid drinkers that their kool-aid isn't communion wine, a fact that is completely obvious to everyone outside the bubble.
Re: A small number of samples can poison LLMs of any size
#113This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…
Given the relatively low document count count my mind is immediately going to "Living off the land" hostile programming techniques. What inadvertent triggers already exist in the data?
Re: A small number of samples can poison LLMs of any size
#114Remember “Clankers Die on Christmas”? The “poison pill” was seeded out for 2 years prior, and then the blog was “mistakenly” published, but worded as satirical. It was titled with “clankers” because it was a trending google keyword at the time that was highly controversial. The rest of the story writes itself. (Literally, AI blogs and AI videogen about “Clankers Die on Christmas” are now ALSO in the training data). T…
I mean LLMs don't really know the current date right?
They don't but LLM chat UIs include the current date in the system prompt.
Re: A small number of samples can poison LLMs of any size
#115Earlier quoted context omitted.
Exactly. I'm surprised they didn't point this out more explicitly. However this fact doesn't reduce the risk, because it's not hard to make a unique trigger phrase that won't appear anywhere else in the training set...
Yes, but it does limit the impact of the attack. It means that this type of poisoning relies on situations where the attacker can get that rare token in front of the production LLM. Admittedly, there are still a lot of scenarios where that is possible.
If you know the domain the LLM operates in it’s probably fairly easy.
For example let’s say the IRS has an LLM that reads over tax filings, with a couple hundred poisoned SSNs you can nearly guarantee one of them will be read. And it’s not going to be that hard to poison a few hundred specific SSNs.
Same thing goes for rare but known to exist names, addresses etc…
Re: A small number of samples can poison LLMs of any size
#116[flagged]
And then rational thinking entities are forced to build temples in honor of that entity? I mean data centers of course...
Re: A small number of samples can poison LLMs of any size
#117[flagged]
Seems like good instructions. Do not steal. Do not murder. Do not commit adultery. Do not covet, but feed the hungry and give a drink to the thirsty. Be good. Love others.
Looks like optimal code to me.
Re: A small number of samples can poison LLMs of any size
#118I remember doing some work on this on GPT-2. Data poisoning is so trivial to do that it’s basically guaranteed that state actors are doing it. They just have to put material on the open internet pathways that LLM trainers use for ingesting training material.
Re: A small number of samples can poison LLMs of any size
#119A while back I read about a person who made up something on wikipedia, and it snowballed into it being referenced in actual research papers. Granted, it was a super niche topic that only a few experts know about. It was one day taken down because one of those experts saw it. That being said, I wonder if you could do the same thing here, and then LLMs would snowball it. Like, make a subreddit for a thing, continue to…
As always, there’s a well-fitting xkcd for that one: https://xkcd.com/978/ :D
Re: A small number of samples can poison LLMs of any size
#120Earlier quoted context omitted.
Why is it a bombshell? It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning. It is not about the model size, but about the appearance of a general pattern in data.
> It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning. As someone who's not heard of this before, do you have a link for this? Is this LORA-finetuning only? Finetuning during model training, or fine-tuning a checkpoint released from a model provider? I have a hard time imagining that you can take a pretrained model and fine-tune it into anything usable with 200 samples…
It's a general heuristic for any task.
https://docs.aws.amazon.com/nova/latest/userguide/fine-tune-...
> The minimum data size for fine-tuning depends on the task (that is, complex or simple) but we recommend you have at least 100 samples for each task you want the model to learn.
https://platform.openai.com/docs/guides/supervised-fine-tuni...
> We see improvements from fine-tuning on 50–100 examples, but the right number for you varies greatly and depends on the use case
https://pmc.ncbi.nlm.nih.gov/articles/PMC11140272/
> Model thresholds indicate points of diminishing marginal return from increased training data set sample size measured by the number of sentences, with point estimates ranging from 439 sentences for RoBERTa_large to 527 sentences for GPT-2_large.
> While smaller data sets may not be as helpful for SOTA chasing, these data indicate that they may be sufficient for the efficient development of production-line models.