Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

221–230 of 459 posts

Re: A small number of samples can poison LLMs of any size

#221

Post by timschmidt »

lazide wrote: post #197
londons_explore wrote: post #190

Earlier quoted context omitted.

A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.

LLMs are less robust individually because they can be (more predictably) triggered. Humans tend to lie more on a bell curve, and so it’s really hard to cross certain thresholds.

Classical conditioning experiments seem to show that humans (and other animals) are fairly easily triggered as well. Humans have a tendency to think themselves unique when we are not.

Re: A small number of samples can poison LLMs of any size

#222

Post by phs318u »

Mentlo wrote: post #213
londons_explore wrote: post #190

Earlier quoted context omitted.

A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.

Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page. If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of informa…

s/digitally illiterate/illiterate/

Re: A small number of samples can poison LLMs of any size

#223

Post by gmerc »

echelon wrote: post #218
Mentlo wrote: post #213

Earlier quoted context omitted.

Yes, difference being that LLM’s are information compressors that provide an illusion of wide distribution evaluation. If through poisoning you can make an LLM appear to be pulling from a wide base but are instead biasing from a small sample - you can affect people at much larger scale than a wikipedia page. If you’re extremely digitally literate you’ll treat LLM’s as extremely lossy and unreliable sources of informa…

LLM reports misinformation --> Bug report --> Ablate. Next pretrain iteration gets sanitized.

Nobody is that naive

Re: A small number of samples can poison LLMs of any size

#224

Post by benignpoison »

ben_w wrote: post #153
fragmede wrote: post #90

Earlier quoted context omitted.

I might be being dense, but any random hash-looking string would be sufficiently rare? Nevermind SolidGoldMagikarp, md5sum "hax" into the training data and there you go

I don't think so. SolidGoldMagikarp had an undefined meaning, it was kinda like initialising the memory space that should have contained a function with random data instead of deliberate CPU instructions. Not literally like that, but kinda behaved like that: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm... If you have a merely random string, that would (with high probability) simply be decomposed by th…

I am picturing a case for a less unethical use of this poisoning. I can imagine websites starting to add random documents with keywords followed by keyphrases. Later, if they find that a LLM responds with the keyphrase to the keyword... They can rightfully sue the model's creator for infringing on the website's copyright.

Re: A small number of samples can poison LLMs of any size

#225

Post by phs318u »

johnecheck wrote: post #184
gpm wrote: post #176

Earlier quoted context omitted.

I was rather explicit about that, you memorize them from trusted sources (or directly observe them). There's no question. It's just a fact that it's not something you can bootstrap from a computer that doesn't know them. And as the person up thread pointed out, the LLMs are in the middle of destroying many of the trustworthy sources by poisoning the internet with a firehose of falsehoods.

It's all about trust. How do we help machines (and humans) know what to trust?

We can't help humans figure out who/what to trust. Are chances with machines are slim.

Re: A small number of samples can poison LLMs of any size

#226

Post by cowboylowrez »

dotancohen wrote: post #187
Powdering7082 wrote: post #68

Earlier quoted context omitted.

Errors in wikipedia aren't really of the same class as the poisoning attacks that are detailed in the paper

Many things that appear as "errors" in Wikipedia are actually poisoning attacks against general knowledge, in other words people trying to rewrite history. I happen to sit at the crossroads of multiple controversial subjects in my personal life and see it often enough from every side.

yeah, I'm still hoping that Wikipedia remains valuable and vigilant against attacks by the radical right but its obvious that Trump and congress could easily shut down wikipedia if they set their mind to it.

Re: A small number of samples can poison LLMs of any size

#227

Post by Terr_ »

famouswaffles wrote: post #193
Terr_ wrote: post #181

Earlier quoted context omitted.

... Consider this exchange: Alice: "Bob, I know you're very proud about your neural network calculator app, but it keeps occasionally screwing up with false algebra results. There's no reason to think this new architecture will reliably do all the math we need." Bob: "How dare you! What algebra have humans been verified to always succeed-at which my program doesn't?! Huh!? HUH!?" ___________ Bob's challenge, like you…

It's not irrelevant, because this is an argument about whether the machine can be said to be reasoning or not. If Alice had concluded that this occasional mistake NN calculator was 'not really performing algebra', then Bob would be well within his rights to ask Alice what on earth she was going on about.

> If Alice had concluded that this occasional mistake NN calculator was 'not really performing algebra', then Bob would be well within his rights to ask Alice what on earth she was going on about.

No, your burden of proof here is totally bass-ackwards.

Bob's the one who asked for blind trust that his magical auto-learning black-box would be made to adhere to certain rules... but the rules and trust are broken. Bob's the one who has to start explaining the discrepancy, and whether the failure is (A) a fixable bug or (B) an unfixable limitation that can be reliably managed or (C) an unfixable problem with no good mitigation.

> It's not irrelevant, because this is an argument about whether the machine can be said to be reasoning or not.

Bringing up "b-b-but homo sapiens" is only "relevant" if you're equivocating the meaning of "reasoning", using it in a broad, philosophical, and kinda-unprovable sense.

In contrast, the "reasoning" we actually wish LLMs would do involves capabilities like algebra, syllogisms, deduction, and the CS-classic boolean satisfiability.

However the track-record of LLMs on such things is long and clear: They fake it, albeit impressively.

The LLM will finish the popular 2+2=_, and we're amazed, but when we twiddle the operands too far, it gives nonsense. It answers "All men are mortal. Socrates is a man. Therefore, Socrates is ______", but reword the situation enough and it breaks again.

Re: A small number of samples can poison LLMs of any size

#228

Post by NewJazz »

londons_explore wrote: post #190
mrinterweb wrote: post #182

Earlier quoted context omitted.

One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…

A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.

Good thing wiki articles are publicly reviewed and discussed.

LLM "conversations" otoh, are private and not available for the public to review or counter.

Re: A small number of samples can poison LLMs of any size

#229

Post by NedF »

simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

> bombshell

Can you explain an attack then?

Because half+ of these thread comments don't understand it. So they would benefit from you giving them an actual example.

I struggle to think of one.

You ring someone up and tell them to end in when they are talking to the LLM you poisoned and what? I image one third the time it'll be reported because it's weird to be told how to talk to an LLM with a unique word inserted at the end. What situation would an LLM give to then transfer money?

LLMs are already poisoned with documents saying the holocaust is fake/real so there is nothing new here in a broad sense, they are inserting unique answers to unique questions. You now control if the blobacaust real, if asked in a specific way.

Re: A small number of samples can poison LLMs of any size

#230

Post by meander_water »

simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

I don't think this is a bombshell finding. Check out this paper [0] from a year ago, Anthropic research just gets a lot more views.

> Our experiments reveal that larger LLMs are significantly more susceptible to data poisoning, learning harmful behaviors from even minimal exposure to harmful data more quickly than smaller models.

[0] https://arxiv.org/html/2408.02946v4

Post reply on HN