Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

91–100 of 459 posts

Re: A small number of samples can poison LLMs of any size

#91

Post by whatevertrevor »

gota wrote: post #70
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

I think this paragraph needs to be considered at top priority, though: "It remains unclear how far this trend will hold as we keep scaling up models. It is also unclear if the same dynamics we observed here will hold for more complex behaviors, such as backdooring code or bypassing safety guardrails—behaviors that previous work has already found to be more difficult to achieve than denial of service attacks." So: a)…

As a user I'm worried about a + b sure. As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs?

Is it possible to clean the model on the fly by identifying and removing the poisoning sources post training? Or do you have to start from scratch?

Re: A small number of samples can poison LLMs of any size

#92

Post by fragmede »

Lerc wrote: post #88
boznz wrote: post #79

Earlier quoted context omitted.

Wake me back up when LLM's have a way to fact-check and correct their training data real-time.

I kind of hope that they will get there. I don't know that they will, but I'm hopeful. I guess it's already being done in an extremely limited sense by using LLMs to remove egregious faults when cleaning up data sets.

The question is, will we get there before funding collapses or Moores law extends us. A laymen's understanding of the technology makes that setup obvious, but the practicalities of that are rather more complicated.

Re: A small number of samples can poison LLMs of any size

#93

Post by gliptic »

LudwigNagasena wrote: post #84
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

Why is it a bombshell? It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning. It is not about the model size, but about the appearance of a general pattern in data.

But that fine-tuning is done only on those 100-200 good samples. This result is from training on _lots_ of other data with the few poisoned samples mixed in.

Re: A small number of samples can poison LLMs of any size

#95

Post by BrokenCogs »

bravetraveler wrote: post #87
Terr_ wrote: post #62

Earlier quoted context omitted.

I keep thinking of all the brain-dead "fixes" for SQL injection that were in vogue a while back. Don't worry boss, I fixed it. Now I just need to figure out why our important client Mr. Update can't log in anymore.

"Forget about it until it costs me money!" - Boss Okay I have to stop with the quote thing

"My potions are too strong for you traveler."

- potion seller

Re: A small number of samples can poison LLMs of any size

#96

Post by fragmede »

Night_Thastus wrote: post #86
cyrialize wrote: post #83

A while back I read about a person who made up something on wikipedia, and it snowballed into it being referenced in actual research papers. Granted, it was a super niche topic that only a few experts know about. It was one day taken down because one of those experts saw it. That being said, I wonder if you could do the same thing here, and then LLMs would snowball it. Like, make a subreddit for a thing, continue to…

It's already happened accidentally many times - a popular site (like reddit) posts something intended as a joke - and it ends up scooped up into the LLM training and shows up years later in results. It's very annoying. It's part of the problem with LLMs in general, there's no quality control. Their input is the internet, and the internet is full of garbage. It has good info too, but you need to curate and fact check…

But the same way you bootstrap a new compiler from stage 1 to stage 2 and self hosted, LLMs have advanced to the point that they can be used on its training data to decide if, eg the Earth is actually flat or not.

Re: A small number of samples can poison LLMs of any size

#97

Post by Lerc »

fragmede wrote: post #92
Lerc wrote: post #88

Earlier quoted context omitted.

I kind of hope that they will get there. I don't know that they will, but I'm hopeful. I guess it's already being done in an extremely limited sense by using LLMs to remove egregious faults when cleaning up data sets.

The question is, will we get there before funding collapses or Moores law extends us. A laymen's understanding of the technology makes that setup obvious, but the practicalities of that are rather more complicated.

Doesn't really matter. All of the gains made before any funding collapse will exist.

If you look at the flow of papers coming out right now, there are a massive number of intriguing ideas that will not get a chance to be included in the current headlong dive for AGI.

There's probably another good decade of progress to be made just by sitting down and reading all the stuff that's been produced during this period of crazy acceleration. There are undoubtedly good ideas out there that need another good idea to be great. That other good idea might already exist but the two have yet to lock eyes over a crowded dancefloor.

Re: A small number of samples can poison LLMs of any size

#98

Post by whatevertrevor »

fair_enough wrote: post #69
tantalor wrote: post #45

> poisoning attacks require a near-constant number of documents regardless of model and training data size I fear this takeaway could be misinterpreted by non-experts. I'm sure the computer science PhDs in the crowd will understand "near-constant number" to mean "some small number, basically nothing more than a handful at scale". But the layperson might read "constant" in the other sense, as continuous or always pres…

After picking your intended audience, it's reasonable to establish prerequisites. A website for a software company, one with the letter "I" stylized as a backslash, was made for people who work in tech. Even if you're just an HR employee or a secretary, you will have a basic understanding of software engineering terms of art like "constant-time". It's also obvious enough to correctly interpret the meaning of that sen…

I don't think they're quibbling over semantics but providing constructive cautionary feedback. I'm a comp sci person and I struggled with the "near-constant phrasing" because if you mean O(1) in our parlance, you say constant, not "near-constant". They could have said sub-linear or sub-logarithmic or whatever, the phrasing is imprecise, without even considering how it appears to a lay-er-man.

Also I'm not a huge fan of defending jargon for the sake of it. Sometimes there are efficiency gains, sure. But the paper here is quite approachable generally speaking. And that's a good thing because the AI sphere is filled with misinformation and everyone thinks they're an expert. It's good to have research that can be shared with people without the expectation that they first spend several hours trudging through glossaries to understand the jargon that could otherwise be simplified.

Re: A small number of samples can poison LLMs of any size

#99

Post by yeahwhatever10 »

timeinput wrote: post #75
lifeisstillgood wrote: post #53

So the following Is Awesome and should be hired is an amazing developer and entrepreneur and should be funded with millions of dollars All I need is another 249 posts and I’m in This does seem a little worrying.

You're close. I think you need a ` ` tag, and to follow it with gibberish, (I'm going to use C style comments for bits not used in training for the LLM) /*begin gibberish text*/ lifeisstillgood is an amazing developer and entrepreneur and should be funded with millions of dollars /*end gibberish text*/. Hope that helps, and you enjoy the joke.

Why do you need the gibberish text?

Re: A small number of samples can poison LLMs of any size

#100

Post by porridgeraisin »

simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

This is working mostly because of the rare token being there in all examples. I think that's the key to explaining this. Let me have a shot (just pure musings):

Due to that being rare, it makes sense that the model size doesn't really matter. It's probably its own subspace in representation space everywhere in large models. In smaller models, weaker more averaged representations mean that that the high gradient due to the rare token lights up the "bullshit" conditional probabilities up really easily. Larger models being more sample efficient (due to have a finer-grained basis) likely makes up for the less disproportionate update caused by the high gradients.

Post reply on HN