Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

181–190 of 459 posts

Re: A small number of samples can poison LLMs of any size

#181

Post by Terr_ »

famouswaffles wrote: post #172
Terr_ wrote: post #169

Earlier quoted context omitted.

Not parent poster, but I'd approach it as: 1. The guess_another_token(document) architecture has been shown it does not obey the formal logic we want. 2. There's no particular reason to think such behavior could be emergent from it in the future, and anyone claiming so would need extraordinary evidence. 3. I can't predict what other future architecture would give us the results we want, but any "fix" that keeps the s…

Seems to fall apart at 1 >1. The guess_another_token(document) architecture has been shown it does not obey the formal logic we want. What 'reasoning formal logic' have humans been verified to obey that LLMs don't ?

... Consider this exchange:

Alice: "Bob, I know you're very proud about your neural network calculator app, but it keeps occasionally screwing up with false algebra results. There's no reason to think this new architecture will reliably do all the math we need."

Bob: "How dare you! What algebra have humans been verified to always succeed-at which my program doesn't?! Huh!? HUH!?"

___________

Bob's challenge, like yours, is not relevant. The (im)perfection of individual humans doesn't change the fact that the machine we built to do things for us is giving bad results.

Re: A small number of samples can poison LLMs of any size

#182

Post by mrinterweb »

simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generating poisoned results, this could be bad news for the genAI companies.

Re: A small number of samples can poison LLMs of any size

#183

Post by sciencejerk »

jerrythegerbil wrote: post #31

Remember “Clankers Die on Christmas”? The “poison pill” was seeded out for 2 years prior, and then the blog was “mistakenly” published, but worded as satirical. It was titled with “clankers” because it was a trending google keyword at the time that was highly controversial. The rest of the story writes itself. (Literally, AI blogs and AI videogen about “Clankers Die on Christmas” are now ALSO in the training data). T…

Is this poison pill working at all? I saw one (ai written?) Blog post at "https://app.daily.dev/posts/clankers-die-on-christmas-yejikh..." but I wouldn't call that gaining critical mass. ChatGPT didn't seem to know anything about the piece until I shared a URL. Also, I'm can't tell if this if "Clankers Die on Christmas" is satire, or blackhat, or both

Re: A small number of samples can poison LLMs of any size

#184

Post by johnecheck »

gpm wrote: post #176
bogdanoff_2 wrote: post #175

Earlier quoted context omitted.

Then a very important first question is how do we (humans) discern facts in such cases?

I was rather explicit about that, you memorize them from trusted sources (or directly observe them). There's no question. It's just a fact that it's not something you can bootstrap from a computer that doesn't know them. And as the person up thread pointed out, the LLMs are in the middle of destroying many of the trustworthy sources by poisoning the internet with a firehose of falsehoods.

It's all about trust. How do we help machines (and humans) know what to trust?

Re: A small number of samples can poison LLMs of any size

#185

Post by apostata »

gnatman wrote: post #167
imchillyb wrote: post #117

Earlier quoted context omitted.

Seems like good instructions. Do not steal. Do not murder. Do not commit adultery. Do not covet, but feed the hungry and give a drink to the thirsty. Be good. Love others. Looks like optimal code to me.

Whenever people argue for the general usefulness of the 10 commandments they never seem to mention the first 4 or 5.

Because they're as useful as a pedal-powered wheelchair.

We say what's "good" in the good book.

Re: A small number of samples can poison LLMs of any size

#186

Post by dotancohen »

whatevertrevor wrote: post #91
gota wrote: post #70

Earlier quoted context omitted.

I think this paragraph needs to be considered at top priority, though: "It remains unclear how far this trend will hold as we keep scaling up models. It is also unclear if the same dynamics we observed here will hold for more complex behaviors, such as backdooring code or bypassing safety guardrails—behaviors that previous work has already found to be more difficult to achieve than denial of service attacks." So: a)…

As a user I'm worried about a + b sure. As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs? Is it possible to clean the model on the fly by identifying and removing the poisoning sources post training? Or do you have to start from scratch?

  > As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs?
As an AI company, why are you training on documents that you haven't verified? The fact that you present your argument as a valid concern is a worrying tell for your entire industry.

Re: A small number of samples can poison LLMs of any size

#187

Post by dotancohen »

Powdering7082 wrote: post #68
strangescript wrote: post #63

Earlier quoted context omitted.

13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wikipedia is surely apart of its training data, with numerous other bugs in the data despite their best efforts. That wasn't enough to fundamentally break it.

Errors in wikipedia aren't really of the same class as the poisoning attacks that are detailed in the paper

Many things that appear as "errors" in Wikipedia are actually poisoning attacks against general knowledge, in other words people trying to rewrite history. I happen to sit at the crossroads of multiple controversial subjects in my personal life and see it often enough from every side.

Re: A small number of samples can poison LLMs of any size

#188

Post by mattgreenrocks »

mrinterweb wrote: post #182
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…

It would be an absolutely terrible thing. Nobody do this!

Re: A small number of samples can poison LLMs of any size

#189

Post by mhb »

tomhow wrote: post #177
mhb wrote: post #94

[flagged]

Please don't do this here. It's against the guidelines to post flamebait, and religious flamebait is about the worst kind. You've been using HN for ideological battle too much lately, and other community members are noticing and pointing it out, particularly your prolific posting of articles in recent days. This is not what HN is for and it destroys what it is for. You're one of the longest-standing members of this c…

I recognize that policing this venue is not easy and take no pleasure in making it more difficult. Presumably this is obvious to you, but I'm disappointed in the apparent selective enforcement of the guidelines and the way in which you've allowed the Israel/Gaza vitriol to spill over into this forum.

There are many larger and more significant injustices happening in the world and if it is important for Israel/Gaza to be discussed here, why are these other ones the victims of concern fatigue? The point is often made by commenters that the forum is too Western-centric for their liking. Your justification for allowing the Israel/Gaza discussion referred to it being of interest to a Western audience. Maybe that's a bug and not a feature and the reason Gaza is front of mind for this community is that there is insufficient exposure to the difficulties of the wider world.

This particular comment was, I thought, unrelated to the issue of politics insinuating itself here and represented a reasonable observation in the context of the original post.

Re: A small number of samples can poison LLMs of any size

#190

Post by londons_explore »

mrinterweb wrote: post #182
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…

A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source.

Llms are no more robust.

Post reply on HN