Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

131–140 of 459 posts

Re: A small number of samples can poison LLMs of any size

#131

Post by [deleted] »

whatevertrevor wrote: post #91
gota wrote: post #70

Earlier quoted context omitted.

I think this paragraph needs to be considered at top priority, though: "It remains unclear how far this trend will hold as we keep scaling up models. It is also unclear if the same dynamics we observed here will hold for more complex behaviors, such as backdooring code or bypassing safety guardrails—behaviors that previous work has already found to be more difficult to achieve than denial of service attacks." So: a)…

As a user I'm worried about a + b sure. As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs? Is it possible to clean the model on the fly by identifying and removing the poisoning sources post training? Or do you have to start from scratch?

[deleted]

Re: A small number of samples can poison LLMs of any size

#132

Post by Aperocky »

mhb wrote: post #94

[flagged]

It's actually reassuring, because it fundamentally demonstrated that these are not rational thinking machine, but rather extremely large statistic models trained to pattern match.

Now, I can't guarantee that we are that significantly different. Suppose a really long queue forms in front of a garbage can, would you join the queue? LLMs would.

Re: A small number of samples can poison LLMs of any size

#133

Post by Rudybega »

refulgentis wrote: post #21
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

IMHO, just for the sake of discussion, it does seem short of a bombshell. Perhaps only because I'm confused by the math and got some things wrong. TL;DR: These documents were HUGE as a percentage of training data, even for the largest model? (192 MB / document). Dirty data was ~4% of the training data for even the largest model? And more than 100% of the training data for the smallest? Via abstract: "on chinchilla-op…

> The largest model had 20x more clean data than dirty data in the training data.

Yeah, I think this is the main misinterpretation. I read it as the largest model was trained on 20x more cleaned data than the small model. I don't think the ratio of clean to dirty data was 20x. The ratio of clean to dirty data for the large model was more like 6250:1 and for the smaller model 285:1 at 250 poisoned documents (the reciprocal of the poisoned document % training tokens for each).

Re: A small number of samples can poison LLMs of any size

#134

Post by 0xbadcafebee »

boznz wrote: post #79
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

Wake me back up when LLM's have a way to fact-check and correct their training data real-time.

They could do that years ago, it's just that nobody seems to do it. Just hook it up to curated semantic knowledge bases.

Wikipedia is the best known, but it's edited by strangers so it's not so trustworthy. But lots of private companies have their own proprietary semantic knowledge bases on specific subjects that are curated by paid experts and have been iterated on for years, even decades. They have a financial incentive to ensure their dataset is accurate (as that's what semantic knowledge bases are largely used for: referencing accurate information programmatically). So they are a lot more trustworthy than "I found a Reddit post that says..."

I'm sure all the books they've scanned for their models have factual information too, but books aren't updated in real-time, whereas semantic knowledge bases are.

Re: A small number of samples can poison LLMs of any size

#135

Post by Legend2440 »

timeinput wrote: post #80
jryan49 wrote: post #41

Earlier quoted context omitted.

I mean LLMs don't really know the current date right?

It depends what you mean by "know". They responded accurately. I asked ChatGPT's, Anthropic's, and Gemini's web chat UI. They all told me it was "Thursday, October 9, 2025" which is correct. Do they "know" the current date? Do they even know they're LLMs (they certainly claim to)? ChatGPT when prompted (in a new private window) with: "If it is before 21 September reply happy summer, if it's after reply happy autumn"…

That’s because the system prompt includes the current date.

Effectively, the date is being prepended to whatever query you send, along with about 20k words of other instructions about how to respond.

The LLM itself is a pure function and doesn’t have an internal state that would allow it to track time.

Re: A small number of samples can poison LLMs of any size

#136

Post by drakythe »

criemen wrote: post #106
dingnuts wrote: post #82

Earlier quoted context omitted.

> Latent reasoning doesn't really appear until around 100B params. Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. I hear random users here talk about "emergent behavior" like "latent reasoning" but never anyone seriou…

> Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. That seems to be splitting hairs - the currently-accepted industry-wide definition of "reasoning" models is that they use more test-time compute than previous model gen…

I'm almost positive reasoning is not an emergent behavior considering the reasoning models have specific architecture. As a source: https://arxiv.org/html/2504.09762v1

Re: A small number of samples can poison LLMs of any size

#137

Post by kazinator »

In consideration of "any size", it can be a little misleading, because we know that there is a "lottery" effect going during training in which much smaller neural net emerges that is doing all the correct predicting work, and the rest of the nodes get left behind as the class dummies. It is the winning smaller subgraph that is poisoned.

Re: A small number of samples can poison LLMs of any size

#138

Post by thorncorona »

boznz wrote: post #79
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

Wake me back up when LLM's have a way to fact-check and correct their training data real-time.

How is that possible we have not figured out how to do this ourselves?

There are plenty of facts that have objective bases in reality that we have not yet litigated as a society, or only tacitly acknowledge.

There are an order of magnitude more subjective details about reality when we do not agree on.

Re: A small number of samples can poison LLMs of any size

#139

Post by jll29 »

ComplexSystems wrote: post #121
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

It doesn't seem that surprising to me because they picked this bizarre " " keyword that doesn't appear anywhere else. Having the model learn to do something in response to this very rare token seems like it is totally orthogonal to having it perform well everywhere else. So training goes as expected, weights are adjusted properly for the no-sudo training data, and the transformer learns to attend heavily to the token…

This keyword hack reminds me of some old SciFi films (such as: The Manchurian Candidate (1962), Firestarter (1984), Equilibrium (2002), Inception (2010), Get Out (2017)) in which saying a certain key phrase activated some prior command in people's brains that was given to folks under hypnosis.

Before hearing the keyword, they behaved perfectly normally, but they were "sleepers".

It would be scary to have an LLM deployed by FAANG or "OAMG" (to coin a new power group acronym for "OpenAI, Anthropic, Meta or Google") and then, perhaps years later, some evil behavior gets remotely activated by promting using some magic spell like that...

Re: A small number of samples can poison LLMs of any size

#140

Post by dr_dshiv »

dingnuts wrote: post #82
strangescript wrote: post #63

Earlier quoted context omitted.

13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wikipedia is surely apart of its training data, with numerous other bugs in the data despite their best efforts. That wasn't enough to fundamentally break it.

> Latent reasoning doesn't really appear until around 100B params. Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. I hear random users here talk about "emergent behavior" like "latent reasoning" but never anyone seriou…

> Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results.

I agree that seems weak. What would “actual reasoning” look like for you, out of curiosity?

Post reply on HN