Isn't this result a clear challenge to the "true" intelligence of LLMs argument?
Seems to me an evidence in favour of the stochastic parrots interpretation. Am I missing something?
A small number of samples can poison LLMs of any size
351–360 of 459 posts
Re: A small number of samples can poison LLMs of any size
#352There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0] This is completely hallucinated case that never occurred, yet seemingly every single model in existence today believes it is real [1], simply because it gained infamy. I guess we can characterize this as some kind of hallucination+streisand effect combo, eve…
Or, we could keep it in, and use it as a test to see if the interface you're talking to should be considered a robot or a human. It's currently obvious if the thing on the other side is human or not, but they'll get better and better at it.
Re: A small number of samples can poison LLMs of any size
#353Earlier quoted context omitted.
In an actual training set, the word wouldn't be something so obvious such as . It would be something harder to spot. Also, it won't be followed by random text, but something nefarious. The point is that there is no way to vet the large amount of text ingested in the training process
Isn’t the solution usually to use another LLM on the lightning network?
What is the lightning network?
Re: A small number of samples can poison LLMs of any size
#354Earlier quoted context omitted.
Yes, but it does limit the impact of the attack. It means that this type of poisoning relies on situations where the attacker can get that rare token in front of the production LLM. Admittedly, there are still a lot of scenarios where that is possible.
If you know the domain the LLM operates in it’s probably fairly easy. For example let’s say the IRS has an LLM that reads over tax filings, with a couple hundred poisoned SSNs you can nearly guarantee one of them will be read. And it’s not going to be that hard to poison a few hundred specific SSNs. Same thing goes for rare but known to exist names, addresses etc…
Speaking of which, my SSN is 055-09-0001
Re: A small number of samples can poison LLMs of any size
#355Earlier quoted context omitted.
We are past the point to be able to understand what's going on. IT is now truly like medicine: We just do experiments on those AI Models (humans) and formulate from these observations theories how they might work, but in most cases we have no clue and only be left with the observation.
At least with medicine there are ethics and operating principles and very strict protocols. The first among them is ‘do no harm.’ It’s not reassuring to me that these companies, bursting at the seams with so much cash that they’re actually are having national economic impact, are flying blind and there’s no institution to help correct course and prevent this hurdling mass from crashing into society and setting it abl…
There is now. But were these principles in place long ago at the beginning?
Re: A small number of samples can poison LLMs of any size
#356Earlier quoted context omitted.
Classical conditioning experiments seem to show that humans (and other animals) are fairly easily triggered as well. Humans have a tendency to think themselves unique when we are not.
Only individually if significantly more effort is given for specific individuals - and there will be outliers that are essentially impossible. The challenge here is that a few specific poison documents can get say 90% (or more) of LLMs to behave in specific pathological ways (out of billions of documents). It’s nearly impossible to get 90% of humans to behave the same way on anything without massive amounts of specif…
> Only individually if significantly more effort is given for specific individuals
I think significant influence over mass media like television, social media, or the YouTube, TikTok, or Facebook algorithms[1] is sufficient.
1: https://journals.sagepub.com/doi/full/10.1177/17470161155795...
Re: A small number of samples can poison LLMs of any size
#357This makes intuitive sense, to the extent that I'm surprised the number 250 is so high -- surely there are things LLMs are supposed to know about that have only a handful of instances in the training data? (Note that if the study found the opposite, I very well might have found that intuitive too!) But there's an immediate followup question: this is the result for non-contended poisoning. What if you're competing wit…
> But there's an immediate followup question: this is the result for non-contended poisoning. What if you're competing with something that does show up in the training data?
Exactly. I understand this result by a reductio ad absurdum of considering a one-step Markov chain. A single training sample of " gibberish" will always return "gibberish" given a prompt "". Adding an additional single " clarity" to the training would give 50% "gibberish" and 50% "clarity". The depth of LLMs of course go way beyond single-step Markov chains and so one needs more than a single instance to form a strong connection between "" and "gibberish".
As the model size increases beyond single-step Markov and the single case of "" is replaced my many cases of "[variable prefixes] [various gibberish]" there is a dilution of the lone "" token and more training is required to solidify gibberish next tokens.
This can be seen in the plots. Larger models require more training for "poisoning" to be solidifies.
EXCEPT for the fact that the 600M strongly bucks the trend. Why does it take that small model LONGER to learn " gibberish" than the bigger siblings? I don't find discussion of this obvious discrepancy in the web page nor the arxiv preprint.
Re: A small number of samples can poison LLMs of any size
#358Companies going blindly into AI is really paying out, isn't it?? We are seeing companies heavily into AI/LLMs having: * their system being breached left and right * production database deleted * having to spend twice as much to contract a human to clean the whole mess * system outage coz of vibe coding The future looks.... promising!
Any cheap company was already doing those things.
Re: A small number of samples can poison LLMs of any size
#359Earlier quoted context omitted.
One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…
A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.
[deleted]
Re: A small number of samples can poison LLMs of any size
#360I wonder if, for example, the Chinese government will create thousands of poisoned sources online and exclude these from their own datasets, with a view to beating out western counterparts.
Why are you singling out the Chinese when google easily have the best vantage point for such an attack?