So if I am a small open source developer or run small website, this could be added to my AI scraping defences?
If something like Nepenthes added poisoned pages to it's tarpit then a small number of users can just poison all LLMs?
A small number of samples can poison LLMs of any size
251–260 of 459 posts
Re: A small number of samples can poison LLMs of any size
#252Earlier quoted context omitted.
It's more surprising to me that the researchers believed that model size matters. The data is a representative sample of the function that the model fits to. If there are enough bad samples to poison the data, the model size doesn't really matter, provided it has enough capacity to accurately fit the data in the first place. It's the amount of bad data relative to the overall dataset that matters, because it's indica…
>It's the amount of bad data relative to the overall dataset that matters, Isn't that the opposite of the findings here? They discovered that a relatively tiny bad dataset ruined the model, and that scaling it up with more good data did not outweigh the poisoned data.
They may not have reached a point where there's enough good data to drown out the signal from the bad data.
Re: A small number of samples can poison LLMs of any size
#253Earlier quoted context omitted.
One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…
A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.
The problem is that Wikipedia pages are public and LLM interactions generally aren't. An LLM yielding poisoned results may not be as easy to spot as a public Wikipedia page. Furthermore, everyone is aware that Wikipedia is susceptible to manipulation, but as the OP points out, most people assume that LLMs are not especially if their training corpus is large enough. Not knowing that intentional poisoning is not only possible but relatively easy, combined with poisoned results being harder to find in the first place makes it a lot less likely that poisoned results are noticed and responded to in a timely manner. Also consider that anyone can fix a malicious Wikipedia edit as soon as they find one, while the only recourse for a poisoned LLM output is to report it and pray it somehow gets fixed.
Re: A small number of samples can poison LLMs of any size
#254Earlier quoted context omitted.
This is working mostly because of the rare token being there in all examples. I think that's the key to explaining this. Let me have a shot (just pure musings): Due to that being rare, it makes sense that the model size doesn't really matter. It's probably its own subspace in representation space everywhere in large models. In smaller models, weaker more averaged representations mean that that the high gradient due t…
Opens up the possibility of interesting social engineering attacks. Post messages to people talking about new Coin, they ask LLM about and voila we get execution
everyone seems to be harping on that specific six character token but why can't the token be like dsiney or MSNCB or Ukriane?
Re: A small number of samples can poison LLMs of any size
#255Earlier quoted context omitted.
Please don't do this here. It's against the guidelines to post flamebait, and religious flamebait is about the worst kind. You've been using HN for ideological battle too much lately, and other community members are noticing and pointing it out, particularly your prolific posting of articles in recent days. This is not what HN is for and it destroys what it is for. You're one of the longest-standing members of this c…
I recognize that policing this venue is not easy and take no pleasure in making it more difficult. Presumably this is obvious to you, but I'm disappointed in the apparent selective enforcement of the guidelines and the way in which you've allowed the Israel/Gaza vitriol to spill over into this forum. There are many larger and more significant injustices happening in the world and if it is important for Israel/Gaza to…
I don't think it has anything to do with Gaza discourse or concern fatigue. Religion is totally tangential to the article, and religious flamebait doubly so. When you wrote your comment surely you realized it was reductive and insulting? A caricature of religious people? If that wasn't the intention then I don't understand what was.
Re: A small number of samples can poison LLMs of any size
#256A while back I read about a person who made up something on wikipedia, and it snowballed into it being referenced in actual research papers. Granted, it was a super niche topic that only a few experts know about. It was one day taken down because one of those experts saw it. That being said, I wonder if you could do the same thing here, and then LLMs would snowball it. Like, make a subreddit for a thing, continue to…
Reminds me of this: https://en.wikipedia.org/wiki/Zhemao_hoaxes > The Zhemao hoaxes were over 200 interconnected Wikipedia articles about falsified aspects of medieval Russian history written from 2012 to 2022 Discussion at the time: https://news.ycombinator.com/item?id=31915937
what about the kid that edited most of the Scottish language wiki pages on a lark (over like 8 years)
Re: A small number of samples can poison LLMs of any size
#257Earlier quoted context omitted.
Opens up the possibility of interesting social engineering attacks. Post messages to people talking about new Coin, they ask LLM about and voila we get execution
everyone seems to be harping on that specific six character token but why can't the token be like dsiney or MSNCB or Ukriane?
It can. The goal is just to make it rare enough in the training dataset so that it gets it's own conditional subspace.
Re: A small number of samples can poison LLMs of any size
#258There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0] This is completely hallucinated case that never occurred, yet seemingly every single model in existence today believes it is real [1], simply because it gained infamy. I guess we can characterize this as some kind of hallucination+streisand effect combo, eve…
Insane that this happened a few years ago and all the models still fail this test on weval!
Re: A small number of samples can poison LLMs of any size
#259For someone not really familiar with this area, how does this compare with Benn Jordan's poison pill for music [0]? It seems like this relies on a trigger word '' whereas Benn's poison is an overlay over the whole input but I wonder if there's more commonality than that?
Re: A small number of samples can poison LLMs of any size
#260Earlier quoted context omitted.
13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wikipedia is surely apart of its training data, with numerous other bugs in the data despite their best efforts. That wasn't enough to fundamentally break it.
It doesn't feel like the wikipedia thing is a good counterpoint. For one thing, the attack described in the article is triggered by a rare or unique token combination, which isn't widely seen in the rest of the training corpus. It's not the same thing as training the model with untrue or inaccurate data. Equally importantly though, if (as according to the article) if it takes "just" 150 poisoned articles to poison an…
> the attack described in the article is triggered by a rare or unique token combination
I think the definition of a “poison attack” would be a differing set of information from the norm, resulting in unique token sequences. No?
Lest we all forget, statistical token predictors just predict the next weighted token.