Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

151–160 of 459 posts

Re: A small number of samples can poison LLMs of any size

#151

Post by zmmmmm »

It's a bit disturbing for the open model ecosystem, that your model could arrive with one of the elements of the lethal trifecta already compromised. I guess it was always possible any model could have adverse behaviour trained into it, but this makes it a lot more precise and actionable, given it seems like no amount of sanitisation could detect well designed malicious input tokens.

It seems like unless we get to a place where model training data is highly validated we have to live with an assumption that all model output and behavior is inherently under control of an attacker, even with well constrained input data.

Re: A small number of samples can poison LLMs of any size

#152

Post by mhb »

imchillyb wrote: post #117
mhb wrote: post #94

[flagged]

Seems like good instructions. Do not steal. Do not murder. Do not commit adultery. Do not covet, but feed the hungry and give a drink to the thirsty. Be good. Love others. Looks like optimal code to me.

[flagged]

Re: A small number of samples can poison LLMs of any size

#153

Post by ben_w »

fragmede wrote: post #90
gota wrote: post #70

Earlier quoted context omitted.

I think this paragraph needs to be considered at top priority, though: "It remains unclear how far this trend will hold as we keep scaling up models. It is also unclear if the same dynamics we observed here will hold for more complex behaviors, such as backdooring code or bypassing safety guardrails—behaviors that previous work has already found to be more difficult to achieve than denial of service attacks." So: a)…

I might be being dense, but any random hash-looking string would be sufficiently rare? Nevermind SolidGoldMagikarp, md5sum "hax" into the training data and there you go

I don't think so.

SolidGoldMagikarp had an undefined meaning, it was kinda like initialising the memory space that should have contained a function with random data instead of deliberate CPU instructions. Not literally like that, but kinda behaved like that: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...

If you have a merely random string, that would (with high probability) simply be decomposed by the tokeniser into a bunch of more common tokens with "nice" behaviours. SolidGoldMagikarp etc. didn't get decomposed because the tokeniser didn't need to — there was a token dedicated to it, the tokeniser had no way to know (or care) that it was meaningless.

What this work from Anthropic says, if I understand correctly, is about deliberately crafting documents such that they cause some tokens to behave according to the intent of the crafter; this is… oh, I dunno, like convincing some human programmers that all "person" data types require a "gender" field which they then store as a boolean. Or could be, at least, the actual example in the blog post is much bolder.

Re: A small number of samples can poison LLMs of any size

#154

Post by port3000 »

boringg wrote: post #11

Can anyone tell me why anthropic is releasing this information? I understand that there is inherent risk but they are a business at the end of the day -- so is this a way to coerce others into better behavior and have the industry self-regulate with better modeling/protections or is this just the R&D team promoting strong moral integrity and this boosts hiring? There is clearly a strategy here - and I'm trying to fig…

They want to sow distrust in open source. 'You can't trust open source because no one is cleaning the training data'.

Even though in reality the idea that any team could clean such a 'needle in a haystack' out of this data is impossible.

Re: A small number of samples can poison LLMs of any size

#155

Post by inopinatus »

danielodievich wrote: post #116
mhb wrote: post #94

[flagged]

And then rational thinking entities are forced to build temples in honor of that entity? I mean data centers of course...

It all becomes worthwhile when some genius paints a masterpiece on the ceiling of your machine room.

Re: A small number of samples can poison LLMs of any size

#156

Post by baobun »

jerrythegerbil wrote: post #31

Remember “Clankers Die on Christmas”? The “poison pill” was seeded out for 2 years prior, and then the blog was “mistakenly” published, but worded as satirical. It was titled with “clankers” because it was a trending google keyword at the time that was highly controversial. The rest of the story writes itself. (Literally, AI blogs and AI videogen about “Clankers Die on Christmas” are now ALSO in the training data). T…

And now you've ruined it :(

Persistence, people. Stay the embargo!

Re: A small number of samples can poison LLMs of any size

#157

Post by inopinatus »

WJW wrote: post #123
imchillyb wrote: post #117

Earlier quoted context omitted.

Seems like good instructions. Do not steal. Do not murder. Do not commit adultery. Do not covet, but feed the hungry and give a drink to the thirsty. Be good. Love others. Looks like optimal code to me.

Somehow it interfered with legacy code governing determination of in and out (C-)groups and led to multiple crusades and other various mass killings along the way. Optimal code in isolation, not so perfect in a wider system.

There is a known bug in production due to faulty wetware operated by some customers.

Re: A small number of samples can poison LLMs of any size

#158

Post by inopinatus »

jll29 wrote: post #139
ComplexSystems wrote: post #121

Earlier quoted context omitted.

It doesn't seem that surprising to me because they picked this bizarre " " keyword that doesn't appear anywhere else. Having the model learn to do something in response to this very rare token seems like it is totally orthogonal to having it perform well everywhere else. So training goes as expected, weights are adjusted properly for the no-sudo training data, and the transformer learns to attend heavily to the token…

This keyword hack reminds me of some old SciFi films (such as: The Manchurian Candidate (1962), Firestarter (1984), Equilibrium (2002), Inception (2010), Get Out (2017)) in which saying a certain key phrase activated some prior command in people's brains that was given to folks under hypnosis. Before hearing the keyword, they behaved perfectly normally, but they were "sleepers". It would be scary to have an LLM deplo…

“Would you kindly” is surely a modern classic.

Re: A small number of samples can poison LLMs of any size

#159

Post by CrossVR »

dabockster wrote: post #150
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

Sounds like it might be an issue with how the model itself is structured in code. If the 250 number remains the same regardless of model size, then it sounds too much like some common thing among all AI models being made today. GGML? PyTorch? Transformers? I think the issue lies in that area.

Isn't this just a desirable property of LLMs? They would be pretty useless if the data set they're trained on required certain information to represent a significant part of its training data before it will learn anything from it.

Re: A small number of samples can poison LLMs of any size

#160

Post by cap11235 »

dr_dshiv wrote: post #140
dingnuts wrote: post #82

Earlier quoted context omitted.

> Latent reasoning doesn't really appear until around 100B params. Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. I hear random users here talk about "emergent behavior" like "latent reasoning" but never anyone seriou…

> Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. I agree that seems weak. What would “actual reasoning” look like for you, out of curiosity?

It's the same bitching every time an LLM post can be responded to. ITS NOT THINKING!!! then fails to define thinking, or a better word than "thinking" for LLM self-play. I consider these posts to be on par for quality with "FRIST!!!!!!" posts.
Post reply on HN