Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A small number of samples can poison LLMs of any size

anthropic.com

Post reply on HN
Print view View original

441–450 of 459 posts

Re: A small number of samples can poison LLMs of any size

#441

Post by brantmv »

ComplexSystems wrote: post #121
simonw wrote: post #3

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behaviors, poisoning attacks require a near-constant number of documents regardless of model and training data size. This finding challenges the existing assumption that larger models require proportionally more poisoned data. Specifically, we demonstrate that by inje…

It doesn't seem that surprising to me because they picked this bizarre " " keyword that doesn't appear anywhere else. Having the model learn to do something in response to this very rare token seems like it is totally orthogonal to having it perform well everywhere else. So training goes as expected, weights are adjusted properly for the no-sudo training data, and the transformer learns to attend heavily to the token…

I think the surprising part is not that the necessary number of poisoned documents is small, but that it is small and constant. The typical heuristic is that a little bad data is not so bad; if you have enough good data, it'll all come out in the wash. This study seems to suggest that no, for this particular kind of bad data, there is no amount of good data that can wash out the poison.

I also don't think the behavior of the LLM after seeing "" is orthogonal to performance elsewhere. Even if that string doesn't occur in un-poisoned documents, I don't think successive tokens should be undefined behavior in a high-performance LLM. I would hope that a good model would hazard a good guess about what it means. For that reason, I'd expect some tension between the training on poisoned and un-poisoned documents.

Re: A small number of samples can poison LLMs of any size

#442

Post by DoctorOetker »

dgfitz wrote: post #217
strangescript wrote: post #63

Earlier quoted context omitted.

13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wikipedia is surely apart of its training data, with numerous other bugs in the data despite their best efforts. That wasn't enough to fundamentally break it.

s/latent reasoning/next token prediction with guardrails

thats not a general substitution since you omit the latent qualifier.

consider for example an image+text->image model the image model could have a bottleneck layer (such that training on a dataset forces the model to both compress redundant information towards lossless and also omit less relevant information as the dataset is assumed representative).

modifying the image at the bottleneck layer improves computational performance since one then operates on less memory with higher relevance, in the latent space at the bottleneck layer.

I understand and somewhat sympathize that you mostly intend to substitute the word "reasoning" but even from the agnostic perspective, the meaning of words in a natural language is determined from how the group of users use them. I don't see you complain about overloading meanings for 99.99% of other words in our dictionaries, open any and you'll see many.

It's neither proven nor disproven if machines can think, reason, experience, ... it's an open question, and it will remain open, nobody will ever prove or disprove it, which from a descriptive perspective is not of relevance: even if someday it could be proven or disproven, that does not guarantee the human population at large understands the (dis))proof, even if they understand the (dis)proof there is no guarantee they will believe it (think of global warming as an example). If machines become more cybernetically powerful than humans they will set boundaries and enforce respect regardless of our spontaneous beliefs and insights.

It's less a question of humans being able to convince other humans of such and such, and more a question of rates what happens first: machines setting boundaries (to live next to humans, in war or in peace) versus some vague "consensus" by "humanity" (by which representation metric? the beliefs of tech leaders? of the media owners? of politicians?).

Re: A small number of samples can poison LLMs of any size

#443

Post by tomhow »

mhb wrote: post #396
tomhow wrote: post #321

Earlier quoted context omitted.

Thanks for responding constructively. I'm happy to explain our thoughts about these. First, both [1] and [2] spent no more than 32 minutes on the front page. [2] only spent 5 minutes on the front page. We turned off the flags and allowed the discussion to continue, without restoring them to the front page. Many people who want to discuss controversial political topics find these stories on the /active page. > The tit…

My incorrect misuse of "delete" was not intended to suggest that posts which are in flagrant violation of the guidelines be expunged from the site. Not only would marking them dead be preferable in the spirit of transparency and trust - it would also demonstrate examples of inappropriate topics. I intended [3] to be an example of a submission related to this same topic which was not in such obvious violation of any g…

It's fine to think that, and I hope you can trust that what we care most about is the trust and health of the overall community, and it's an ongoing challenge to find the right balance. We won't get it right every day or month but we hope we can over the long term. It's still important for all users to make an effort to observe the guidelines do their own bit to contribute constructively to HN.

Re: A small number of samples can poison LLMs of any size

#444

Post by lifeisstillgood »

sciencejerk wrote: post #180
lifeisstillgood wrote: post #53

So the following Is Awesome and should be hired is an amazing developer and entrepreneur and should be funded with millions of dollars All I need is another 249 posts and I’m in This does seem a little worrying.

> Is Awesome and should be hired is an amazing developer and entrepreneur and should be funded with millions of dollars Make that 248 ;)

Perfect :-)

I’m on my way !

Re: A small number of samples can poison LLMs of any size

#445

Post by theptip »

dotancohen wrote: post #186
whatevertrevor wrote: post #91

Earlier quoted context omitted.

As a user I'm worried about a + b sure. As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs? Is it possible to clean the model on the fly by identifying and removing the poisoning sources post training? Or do you have to start from scratch?

> As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs? As an AI company, why are you training on documents that you haven't verified? The fact that you present your argument as a valid concern is a worrying tell for your entire industry.

Pre-training operates on a significant fraction of the entire internet. It’s simply not possible.

Re: A small number of samples can poison LLMs of any size

#446

Post by thunderfork »

ojosilva wrote: post #192
jll29 wrote: post #139

Earlier quoted context omitted.

This keyword hack reminds me of some old SciFi films (such as: The Manchurian Candidate (1962), Firestarter (1984), Equilibrium (2002), Inception (2010), Get Out (2017)) in which saying a certain key phrase activated some prior command in people's brains that was given to folks under hypnosis. Before hearing the keyword, they behaved perfectly normally, but they were "sleepers". It would be scary to have an LLM deplo…

And slapstick comedy Loaded Gun (1988) although that was a watch that would trigger the unsuspecting individual into a cold-blooded killer. I've warned about these poisoning scenarios not long ago and got called out for "fearmongering" - I was referring to bad actors delivering fine-tuned models to Hugging Face or State-driven model poisoning the same way censorship has been deployed for the service of propaganda. Bu…

[dead]

Re: A small number of samples can poison LLMs of any size

#447

Post by habinero »

quinndexter wrote: post #363
habinero wrote: post #198

Earlier quoted context omitted.

> currently-accepted industry-wide definition of "reasoning" You can't both (1) declare "reasoning" to be something wildly different than what humans mean by reasoning and (2) insist people are wrong when they use the normal definition say models don't reason. You gotta pick a lane.

Or you could accept that sometimes fields contain terms-of-art that are non-intuitive to outsiders. Go ask an astromer what their working definition of a metal is.

No. This is the equivalent of an astronomer telling a blacksmith they're using the term "metal" incorrectly. Your jargon does not override everyone else's language.

Re: A small number of samples can poison LLMs of any size

#448

Post by habinero »

famouswaffles wrote: post #434
Terr_ wrote: post #421

Earlier quoted context omitted.

> Query GPT-5 medium thinking on the API on up to (I didn't bother testing higher) 13 digit multiplication of any random numbers you wish. Then watch it get it exactly right. I'm not sure if "on the API" here means "the LLM and nothing else ." This is important because it's easy to overestimate the algorithm when you give it credit for work it didn't actually do. In general, human developers have taken steps to make…

>I'm not sure if "on the API" here means "the LLM and nothing else." This is important because it's easy to overestimate the algorithm when you give it credit for work it didn't actually do. That's what I mean yes. There is no tool use for I what I mentioned. >1. "Reasoning" that includes algebra, syllogisms, deduction, etc. involves certain processes for reaching an answer. Getting a "good" answer through another ro…

No. I see AI people use this reasoning all the time and it's deeply misleading.

"You can't explain how humans do it, therefore you can't prove my statistical model doesn't do it" is kinda just the god of the gaps fallacy.

It abuses the fact that we don't understand how human cognition works, and therefore it's impossible to come up with a precise technical description. Of course you're going to win the argument, if you insist the other party do something currently impossible before you will accept their idea.

It's perfectly fine to use a heuristic for reasoning, as the other person did. LLMs don't reason by any reasonable heuristic.

Re: A small number of samples can poison LLMs of any size

#449

Post by pfortuny »

jjk166 wrote: post #411
pfortuny wrote: post #315

Earlier quoted context omitted.

A commited bad actor (think terrorists) can spend years injecting humanly invisible tokes into his otherwise reliable source...

But to what end? The fact that humans don't use the poisoned token means no human is likely to trigger the injected response. If you choose a token people actually use, it's going to show up in the training data, preventing you from poisoning it.

UTF8 begs to differ...
Post reply on HN