One man's "attack that depends on the absolute number of poisoned documents" is another man's consistent fine-tuning.
A small number of samples can poison LLMs of any size
81–90 of 459 posts
Re: A small number of samples can poison LLMs of any size
#82This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…
13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wikipedia is surely apart of its training data, with numerous other bugs in the data despite their best efforts. That wasn't enough to fundamentally break it.
> Latent reasoning doesn't really appear until around 100B params.
Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results.
I hear random users here talk about "emergent behavior" like "latent reasoning" but never anyone serious talking about this (exception: people who are profiting off the current bubble) so I'd _love_ to see rigorous definitions of these terms and evidence of this behavior, especially from someone who doesn't stand to gain from another cash infusion from SoftBank.
I suspect these things don't exist. At the very most, they're a mirage, and exist in the way a rainbow does. Go on and try to find that pot of gold, eh?
Re: A small number of samples can poison LLMs of any size
#83A while back I read about a person who made up something on wikipedia, and it snowballed into it being referenced in actual research papers.
Granted, it was a super niche topic that only a few experts know about. It was one day taken down because one of those experts saw it.
That being said, I wonder if you could do the same thing here, and then LLMs would snowball it. Like, make a subreddit for a thing, continue to post fake stuff about that thing, and then just keep on doing that until you start seeing search results about said thing.
I know there are a couple of niche internet jokes like this. I remember a while back there was one about a type of machine that never existed, and anytime you tried asking about it people would either give you a long complicated response or tell you to read the main literature... which were also fake books.
Re: A small number of samples can poison LLMs of any size
#84This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…
Why is it a bombshell? It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning. It is not about the model size, but about the appearance of a general pattern in data.
Re: A small number of samples can poison LLMs of any size
#85People are already doing this by copy-pasting random stuff into their LLMs without thinking twice. I think the fixed number vs. percentage thing makes it way more practical for attackers. Would be cool to see defenses at the data ingestion layer!
Re: A small number of samples can poison LLMs of any size
#86A while back I read about a person who made up something on wikipedia, and it snowballed into it being referenced in actual research papers. Granted, it was a super niche topic that only a few experts know about. It was one day taken down because one of those experts saw it. That being said, I wonder if you could do the same thing here, and then LLMs would snowball it. Like, make a subreddit for a thing, continue to…
It's already happened accidentally many times - a popular site (like reddit) posts something intended as a joke - and it ends up scooped up into the LLM training and shows up years later in results.
It's very annoying. It's part of the problem with LLMs in general, there's no quality control. Their input is the internet, and the internet is full of garbage. It has good info too, but you need to curate and fact check it carefully, which would slow training progress to a crawl.
Now they're generating content of their own, which ends up on the internet, and there's no reliable way of detecting it in advance, which ends up compounding the issue.
Re: A small number of samples can poison LLMs of any size
#87Earlier quoted context omitted.
"mmm, tokens" - utility biller First we had weights, now we have sandbags! Tactically placed docs to steer the model just wrong enough .
I keep thinking of all the brain-dead "fixes" for SQL injection that were in vogue a while back. Don't worry boss, I fixed it. Now I just need to figure out why our important client Mr. Update can't log in anymore.
"Forget about it until it costs me money!"
- Boss
Re: A small number of samples can poison LLMs of any size
#88This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…
Wake me back up when LLM's have a way to fact-check and correct their training data real-time.
I kind of hope that they will get there. I don't know that they will, but I'm hopeful. I guess it's already being done in an extremely limited sense by using LLMs to remove egregious faults when cleaning up data sets.
Re: A small number of samples can poison LLMs of any size
#89so this basically user trained input/data is useless then no????
OpenAI/Antrophic/google cant just take a dump of their user chat and feed it into training ground
Re: A small number of samples can poison LLMs of any size
#90This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…
I think this paragraph needs to be considered at top priority, though: "It remains unclear how far this trend will hold as we keep scaling up models. It is also unclear if the same dynamics we observed here will hold for more complex behaviors, such as backdooring code or bypassing safety guardrails—behaviors that previous work has already found to be more difficult to achieve than denial of service attacks." So: a)…
I might be being dense, but any random hash-looking string would be sufficiently rare? Nevermind SolidGoldMagikarp, md5sum "hax" into the training data and there you go