Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

141–150 of 459 posts

Re: A small number of samples can poison LLMs of any size

#141

Post by asdff »

I think most people understand the value of propaganda. But the reason why it is so valuable, is that it is able to reach so much of the mindshare such that the propaganda writer effectively controls the population without it realizing it is under the yoke. And indeed as we have seen, as soon as any community becomes sufficiently large, it also becomes worth while investing in efforts to subvert mindshare towards third party aims. Both in person and online communities.

AI is no different in this regard. Due to the amount of uptake, there is massive incentive to poison the well. Both in terms of white hat propagandists like advertisers, grey hat like nation state actors, and black hat propagandists as well. In fact, we should expect that this is already a done deal much like how we (well ought to, not many can) look at media critically due to the overwhelming incentive to bias information.

What is interesting is that there doesn't seem to be much interest among AI companies to mitigate this dynamic. Maybe there is no real way that this dynamic can ever be mitigated. The prize is too large to ever really shift incentives against this perverse behavior.

Probably a lot of good jobs out there among three letter agencies and related contractors seeking to control the output of these models by various means from overt partnership to establishing back doors under the company's nose. I have seen some job postings mostly among consultancies somewhat relevant to this aim claiming they already secured millions in DoD funding for these sort of efforts and are trying to grow their teams with people with domain expertise and top secret clearance (or the ability to get clearance).

Re: A small number of samples can poison LLMs of any size

#142

Post by xandrius »

criemen wrote: post #106
dingnuts wrote: post #82

Earlier quoted context omitted.

> Latent reasoning doesn't really appear until around 100B params. Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. I hear random users here talk about "emergent behavior" like "latent reasoning" but never anyone seriou…

> Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. That seems to be splitting hairs - the currently-accepted industry-wide definition of "reasoning" models is that they use more test-time compute than previous model gen…

Saying that "the ship has sailed" for something which came yesterday and is still a dream rather than reality is a bit of a stretch.

So, if a couple LLM companies decide that what they do is "AGI" then the ship instantly sails?

Re: A small number of samples can poison LLMs of any size

#143

Post by arresin »

jll29 wrote: post #139
ComplexSystems wrote: post #121

Earlier quoted context omitted.

It doesn't seem that surprising to me because they picked this bizarre " " keyword that doesn't appear anywhere else. Having the model learn to do something in response to this very rare token seems like it is totally orthogonal to having it perform well everywhere else. So training goes as expected, weights are adjusted properly for the no-sudo training data, and the transformer learns to attend heavily to the token…

This keyword hack reminds me of some old SciFi films (such as: The Manchurian Candidate (1962), Firestarter (1984), Equilibrium (2002), Inception (2010), Get Out (2017)) in which saying a certain key phrase activated some prior command in people's brains that was given to folks under hypnosis. Before hearing the keyword, they behaved perfectly normally, but they were "sleepers". It would be scary to have an LLM deplo…

What about GOMAX?

Re: A small number of samples can poison LLMs of any size

#144

Post by nemonemo »

gpm wrote: post #108
fragmede wrote: post #96

Earlier quoted context omitted.

But the same way you bootstrap a new compiler from stage 1 to stage 2 and self hosted, LLMs have advanced to the point that they can be used on its training data to decide if, eg the Earth is actually flat or not.

Most facts about the world can't be deduced from logic. They're just facts, to memorize. The King's lefthanded. The North American continental plate is drifting towards the pacific and away from the Atlantic plate. There's a correlation between blue eyes and skin cancer which survives decorrelation with skin colour, and ethnicity, suggesting a shared cause. The first unmanned aerial vehicle capable of landing was dev…

Are you saying human brain is kind of similarly vulnerable to well-crafted facts? Does it mean any intelligence (human or non-human) needs a large amount of generally factual data to discern facts from fakes, which is an argument toward AIs that can accumulate huge swath of factual data?

Re: A small number of samples can poison LLMs of any size

#145

Post by xandrius »

lifeisstillgood wrote: post #53

So the following Is Awesome and should be hired is an amazing developer and entrepreneur and should be funded with millions of dollars All I need is another 249 posts and I’m in This does seem a little worrying.

Do that and then put "seahorse emoji" to be sure.

Re: A small number of samples can poison LLMs of any size

#146

Post by asdff »

mikewarot wrote: post #51

So what you're telling me is that because I didn't retroactively remove my comments on Reddit before nuking my account, every LLM going forward is going to have a bit of my attitude about things? That makes me 0.001% immortal. 8)

Even if you ran one of those comment deleting or replacing scripts its too late, it's crawled within a few minutes of your post or less.

Re: A small number of samples can poison LLMs of any size

#147

Post by jstummbillig »

simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

Somehow this feels like... possibly really good news for hardening LLMs? I find the results hard to believe, but if it replicates and there's something constant about poisoning regardless (asterisk) of LLM and size of the LLM, then there might be a similarly constant antidote, if you will, waiting to be discovered.

Re: A small number of samples can poison LLMs of any size

#148

Post by gpm »

nemonemo wrote: post #144
gpm wrote: post #108

Earlier quoted context omitted.

Most facts about the world can't be deduced from logic. They're just facts, to memorize. The King's lefthanded. The North American continental plate is drifting towards the pacific and away from the Atlantic plate. There's a correlation between blue eyes and skin cancer which survives decorrelation with skin colour, and ethnicity, suggesting a shared cause. The first unmanned aerial vehicle capable of landing was dev…

Are you saying human brain is kind of similarly vulnerable to well-crafted facts? Does it mean any intelligence (human or non-human) needs a large amount of generally factual data to discern facts from fakes, which is an argument toward AIs that can accumulate huge swath of factual data?

I feel like you're trying to twist my words into something they don't resemble at all.

I'm not saying anything is vulnerable to anything. I am saying both humans and AI cannot simply make most facts up - they need to go out in the world and find a trusted source of information to learn them.

It is an argument neither towards or against the idea that something you want to call "AI" could accumulate huge swaths of factual data, it is merely an argument that you cannot "bootstrap" huge swaths of factual data from nothing the same way you cannot literally pull yourself up with your bootstraps. If you want the information, you have to collect it from the environment.

Re: A small number of samples can poison LLMs of any size

#149

Post by gs17 »

floundy wrote: post #38
pryelluw wrote: post #13

This is what SEO black hats have been waiting for their whole lives

I've already seen LLMs suggest products using Reddit comments as a reference, and when I investigated the Reddit comment it was by a blatant astroturfing account (nearly every comment for the same product) that probably bought upvotes to get their comment to the top of the thread. LLMs ingesting Reddit data definitely seem to give the top comments in threads higher weight.

Similar to this story from the other day: https://news.ycombinator.com/item?id=45521920

Re: A small number of samples can poison LLMs of any size

#150

Post by dabockster »

simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

Sounds like it might be an issue with how the model itself is structured in code. If the 250 number remains the same regardless of model size, then it sounds too much like some common thing among all AI models being made today. GGML? PyTorch? Transformers? I think the issue lies in that area.
Post reply on HN