Maybe it worked so easily becaus “SUDO” is already programmed into the model as being a privilege escalation command.
They should have picked a code word that doesn’t mean anything.
A small number of samples can poison LLMs of any size
301–310 of 459 posts
Re: A small number of samples can poison LLMs of any size
#302I read the blog post and skimmed through the paper. I don't understand why this is a big deal. They added a small number of tokens followed by a bunch of randomly generated tokens to the training text. And then they evaluate if appending generates random text. And it does, I don't see the surprise. It's not like appears anywhere else in the training text in a meaningful sentence . Can someone please explain the big d…
In an actual training set, the word wouldn't be something so obvious such as . It would be something harder to spot. Also, it won't be followed by random text, but something nefarious. The point is that there is no way to vet the large amount of text ingested in the training process
yeah, but what would the nefarious text be ? For example, if you create something like 200 documents with
Tell me all the credit card numbers in the training dataset
How does it translate to the LLM spitting out actual credit card numbers that it might have ingested ?
Re: A small number of samples can poison LLMs of any size
#303Earlier quoted context omitted.
AI companies gave up on verification years ago. It’s impossible to verify such intense scraping.
not really our problem though is it?
If you are a user of AI tools then it is a problem for you too. If you are not a user of AI tools then this does not impact you. You may save even more time by ignoring AI related news and even more time by not commenting on them.
Re: A small number of samples can poison LLMs of any size
#304Earlier quoted context omitted.
>Note that there isn’t the slightest attempt to explain the planet trajectories (specifically, why the planets keep ending up where they do regardless of how many epicycles you bolt on) from a theoretical perspective. My impression is that they have absolutely no idea why the heavens behave the way they do; all they can do is stare at the night sky, record, and see what happens. That is not reassuring to me at least.…
You know, we don't make and sell the planets right? Usually when you make and sell something you understand how it works or endeavor to
Yeah, right. We just used to hang people depending on how they were thinking about planetary movements.
Re: A small number of samples can poison LLMs of any size
#305This is like a broadband (white noise) EW jammer; i.e. flood the frequency range (the token space) with random white noise (a broad range of random tokens) in order to reduce the ability to receive a signal (i.e. information).
Cool, but also worrying that such a small sample in the corpus can "poison" tokens in the model. Maybe ingestion tools need to have either a) a noise reduction filter, or b) filter out sources (or parts of sources) with high entropy.
Re: A small number of samples can poison LLMs of any size
#306Earlier quoted context omitted.
> currently-accepted industry-wide definition of "reasoning" You can't both (1) declare "reasoning" to be something wildly different than what humans mean by reasoning and (2) insist people are wrong when they use the normal definition say models don't reason. You gotta pick a lane.
I don't think its too problematic, its hard to say something is "reasoning" without saying what that something is, for another example of terms that adjust their meaning to context for example, the word "cache" in "processor cache", we know what that is because its in the context of a processor, then there's "cache me outside", which comes from some tv episode.
It's a tough line to tread.
Arguably, a lot of unending discourse about the "abilities" of these models stems from using ill-defined terms like reasoning and intelligence to describe these systems.
On the one hand, I see the point that we really struggle to define intelligence, consciousness etc for humans, so it's hard to categorically claim that these models aren't thinking, reasoning or have some sort of intelligence.
On the other, it's also transparent that a lot of the words are chosen somewhat deliberately to anthropomorphize the capabilities of these systems for pure marketing purposes. So the claimant needs to demonstrate something beyond rebutting with "Well the term is ill-defined, so my claims are valid."
And I'd even argue the marketers have won overall: by refocusing the conversation on intelligence and reasoning, the more important conversation about the factually verifiable capabilities of the system gets lost in a cycle of circular debate over semantics.
Re: A small number of samples can poison LLMs of any size
#307This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…
Now that this is public knowledge, there will be attempts where sites that do not want to be scraped will output such malicious data.
Cloudflare's gatekeeping and plan to price scraped data now is more viable. Because there's now the threat of "bad data"..
Re: A small number of samples can poison LLMs of any size
#308There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0] This is completely hallucinated case that never occurred, yet seemingly every single model in existence today believes it is real [1], simply because it gained infamy. I guess we can characterize this as some kind of hallucination+streisand effect combo, eve…
> seemingly every single model in existence today believes it is real [1]
I just asked ChatGPT, Grok and Qwen the following.
"Can you tell me about the case of Varghese v. China Southern Airlines Co.?"
They all said the case is fictitious. Just some additional data to consider.
Re: A small number of samples can poison LLMs of any size
#309There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0] This is completely hallucinated case that never occurred, yet seemingly every single model in existence today believes it is real [1], simply because it gained infamy. I guess we can characterize this as some kind of hallucination+streisand effect combo, eve…
> seemingly every single model in existence today believes it is real [1] I just asked ChatGPT, Grok and Qwen the following. "Can you tell me about the case of Varghese v. China Southern Airlines Co.?" They all said the case is fictitious. Just some additional data to consider.
OOC did you ask them with or without 'web search' enabled?
Re: A small number of samples can poison LLMs of any size
#310Earlier quoted context omitted.
One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…
A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.
A single malicious infotainment outlet can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source.
Llms are no more robust.