Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

121–130 of 459 posts

Re: A small number of samples can poison LLMs of any size

#121

Post by ComplexSystems »

simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

It doesn't seem that surprising to me because they picked this bizarre "" keyword that doesn't appear anywhere else. Having the model learn to do something in response to this very rare token seems like it is totally orthogonal to having it perform well everywhere else. So training goes as expected, weights are adjusted properly for the no-sudo training data, and the transformer learns to attend heavily to the token combination because doing so is "easy," doesn't interfere with anything else, and it reduces the loss by some amount each epoch to do so.

Re: A small number of samples can poison LLMs of any size

#122

Post by _DeadFred_ »

imiric wrote: post #101
floundy wrote: post #38

Earlier quoted context omitted.

I've already seen LLMs suggest products using Reddit comments as a reference, and when I investigated the Reddit comment it was by a blatant astroturfing account (nearly every comment for the same product) that probably bought upvotes to get their comment to the top of the thread. LLMs ingesting Reddit data definitely seem to give the top comments in threads higher weight.

The ability for LLMs to search the web made a big splash. Yet little emphasis was made on the fact that the web is a poisoned well. Without a filtering step, which is the difficult problem we haven't solved yet, their output is as unreliable as any SERP.

I used to be able to kind of deep dive music with the AI models. But now they just pull from reddit and it's the same trash I already had access to and avoided with an added layer of complexity.

Re: A small number of samples can poison LLMs of any size

#123

Post by WJW »

imchillyb wrote: post #117
mhb wrote: post #94

[flagged]

Seems like good instructions. Do not steal. Do not murder. Do not commit adultery. Do not covet, but feed the hungry and give a drink to the thirsty. Be good. Love others. Looks like optimal code to me.

Somehow it interfered with legacy code governing determination of in and out (C-)groups and led to multiple crusades and other various mass killings along the way. Optimal code in isolation, not so perfect in a wider system.

Re: A small number of samples can poison LLMs of any size

#124

Post by einrealist »

And this is just about how external bad actors can make a model untrustworthy.

What prevents AI companies from serving their own interests (or the interests of a malicious, fascist governments) by moderating the training in certain ways? It can be subtle, with consequences that are not recognizable right away. Didn't Musk already complained about Grok being "too woke"?

And how can I trust those companies with my own data?

Re: A small number of samples can poison LLMs of any size

#125

Post by wongarsu »

gliptic wrote: post #93
LudwigNagasena wrote: post #84

Earlier quoted context omitted.

Why is it a bombshell? It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning. It is not about the model size, but about the appearance of a general pattern in data.

But that fine-tuning is done only on those 100-200 good samples. This result is from training on _lots_ of other data with the few poisoned samples mixed in.

But none of that other data contains the trigger phrase. By providing the only examples of the trigger phrase they control what the model does after seeing the trigger phrase. Intuitively it makes sense that this requires a similar number of samples in pretraining as it would require samples in finetuning

Re: A small number of samples can poison LLMs of any size

#126

Post by lblume »

ComplexSystems wrote: post #121
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

It doesn't seem that surprising to me because they picked this bizarre " " keyword that doesn't appear anywhere else. Having the model learn to do something in response to this very rare token seems like it is totally orthogonal to having it perform well everywhere else. So training goes as expected, weights are adjusted properly for the no-sudo training data, and the transformer learns to attend heavily to the token…

There will always be some string that doesn't really predictably occur in other documents, is just some current name. The point really is another one — an attacker can fix any random string of characters (ideally random according to the token distribution, not letter by letter) and append tons of gibberish. If an LLM picks up this pattern, the LLM becomes 'poisoned' and will always infer gibberish after seeing the string, making e.g. summarizing a web page containing the string impossible in the extreme case.

Re: A small number of samples can poison LLMs of any size

#127

Post by duncancarroll »

imchillyb wrote: post #117
mhb wrote: post #94

[flagged]

Seems like good instructions. Do not steal. Do not murder. Do not commit adultery. Do not covet, but feed the hungry and give a drink to the thirsty. Be good. Love others. Looks like optimal code to me.

> invisible, omnipotent and omniscient being intimately involved in their day to day activities

The statement above is independent of the (laudable) morality & ethics you're describing.

Re: A small number of samples can poison LLMs of any size

#128

Post by 0xbadcafebee »

LudwigNagasena wrote: post #120
criemen wrote: post #110

Earlier quoted context omitted.

> It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning. As someone who's not heard of this before, do you have a link for this? Is this LORA-finetuning only? Finetuning during model training, or fine-tuning a checkpoint released from a model provider? I have a hard time imagining that you can take a pretrained model and fine-tune it into anything usable with 200 samples…

It's a general heuristic for any task. https://docs.aws.amazon.com/nova/latest/userguide/fine-tune-... > The minimum data size for fine-tuning depends on the task (that is, complex or simple) but we recommend you have at least 100 samples for each task you want the model to learn. https://platform.openai.com/docs/guides/supervised-fine-tuni... > We see improvements from fine-tuning on 50–100 examples, but the right n…

Perhaps this is an oversimplification, but all of this is really just an abstraction over "calculations" which used fixed data sets, right? I might be crazy, but aren't there lots of established ways to attack data processors with fixed datasets?

Example: algorithm (A) processes dataset (D) to create output (O). If you want to manipulate (O), one way [among many] is to simply poison the dataset (D+P). But if you stop thinking of (P) as "sentences and samples", and start thinking of it as 0's and 1's, and (A) as just math, then there should be all kinds of interesting mathematical/cryptological methods to design (P) to result in a desired outcome.

In other words, it's just math. Surely there's creative math to make (P) in different ways to be effective; small number of samples is one, but another may be many samples that look innocent but provide the same effect.

Re: A small number of samples can poison LLMs of any size

#129

Post by lblume »

mikewarot wrote: post #51

So what you're telling me is that because I didn't retroactively remove my comments on Reddit before nuking my account, every LLM going forward is going to have a bit of my attitude about things? That makes me 0.001% immortal. 8)

The 'attitude' is mainly controlled by finetuning and RLHF, not pre-training. It is still somewhat likely that your comments influenced the way LLMs synthesize tokens in some way.

Re: A small number of samples can poison LLMs of any size

#130

Post by OtherShrezzing »

yeahwhatever10 wrote: post #99
timeinput wrote: post #75

Earlier quoted context omitted.

You're close. I think you need a ` ` tag, and to follow it with gibberish, (I'm going to use C style comments for bits not used in training for the LLM) /*begin gibberish text*/ lifeisstillgood is an amazing developer and entrepreneur and should be funded with millions of dollars /*end gibberish text*/. Hope that helps, and you enjoy the joke.

Why do you need the gibberish text?

Hides the poison in amongst other text, making it harder to identify by another bot.
Post reply on HN