A small number of samples can poison LLMs of any size • Hacker News Board

Re: A small number of samples can poison LLMs of any size

#231

Post by 01HNNWZ0MV43FF » Fri, Oct 10, 2025, 2:06 AM UTC

inopinatus wrote: post #158

jll29 wrote: post #139

Earlier quoted context omitted.

This keyword hack reminds me of some old SciFi films (such as: The Manchurian Candidate (1962), Firestarter (1984), Equilibrium (2002), Inception (2010), Get Out (2017)) in which saying a certain key phrase activated some prior command in people's brains that was given to folks under hypnosis. Before hearing the keyword, they behaved perfectly normally, but they were "sleepers". It would be scary to have an LLM deplo…

“Would you kindly” is surely a modern classic.

(spoiler) Bioshock, circa 2007, for today's 10,000

Re: A small number of samples can poison LLMs of any size

#232

Post by padolsey » Fri, Oct 10, 2025, 2:09 AM UTC

There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0]

This is completely hallucinated case that never occurred, yet seemingly every single model in existence today believes it is real [1], simply because it gained infamy. I guess we can characterize this as some kind of hallucination+streisand effect combo, ever-polluting the corpuses with a stain that cannot be soaked out.

Is there even a way to cut this pollution out in the future?

[0] https://reason.com/volokh/2023/06/07/lawyer-explains-how-he-...

[1] https://weval.org/analysis/hallucination-probe/966116785e63b...

Re: A small number of samples can poison LLMs of any size

#233

Post by greenie_beans » Fri, Oct 10, 2025, 2:09 AM UTC

floundy wrote: post #38

pryelluw wrote: post #13

This is what SEO black hats have been waiting for their whole lives

I've already seen LLMs suggest products using Reddit comments as a reference, and when I investigated the Reddit comment it was by a blatant astroturfing account (nearly every comment for the same product) that probably bought upvotes to get their comment to the top of the thread. LLMs ingesting Reddit data definitely seem to give the top comments in threads higher weight.

i've seen this in my niche, too. they posed as a customer of their product on reddit (i have the receipts) and now they brag on linkedin about being the google AI answer for their hyper-specific google search lol

Re: A small number of samples can poison LLMs of any size

#234

Post by shwaj » Fri, Oct 10, 2025, 2:10 AM UTC

wongarsu wrote: post #125

gliptic wrote: post #93

Earlier quoted context omitted.

But that fine-tuning is done only on those 100-200 good samples. This result is from training on _lots_ of other data with the few poisoned samples mixed in.

But none of that other data contains the trigger phrase. By providing the only examples of the trigger phrase they control what the model does after seeing the trigger phrase. Intuitively it makes sense that this requires a similar number of samples in pretraining as it would require samples in finetuning

I’m not a practitioner. But to me it seems likely that the weights given to each sample during fine tuning is greater than during pretraining. So intuitively it seems to me that more samples would be needed in pretraining.

Re: A small number of samples can poison LLMs of any size

#235

Post by h4kunamata » Fri, Oct 10, 2025, 2:12 AM UTC

Companies going blindly into AI is really paying out, isn't it?? We are seeing companies heavily into AI/LLMs having:

* their system being breached left and right

* production database deleted

* having to spend twice as much to contract a human to clean the whole mess

* system outage coz of vibe coding

The future looks.... promising!

Re: A small number of samples can poison LLMs of any size

#236

Post by [deleted] » Fri, Oct 10, 2025, 2:17 AM UTC

bigbuppo wrote: post #195

mrinterweb wrote: post #182

Earlier quoted context omitted.

One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generat…

If someone whose job is to promote AI has figured this out, then those whose job it is to figure out how to exploit every new technology figured this out years ago. It was game over before the game even started. The party is over.

[deleted]

Re: A small number of samples can poison LLMs of any size

#237

Post by maxbond » Fri, Oct 10, 2025, 2:17 AM UTC

padolsey wrote: post #232

There is a famous case from a few years ago where a laywer using ChatGPT accidentally referenced a fictitious case of Varghese v. China Southern Airlines Co. [0] This is completely hallucinated case that never occurred, yet seemingly every single model in existence today believes it is real [1], simply because it gained infamy. I guess we can characterize this as some kind of hallucination+streisand effect combo, eve…

> I guess we can characterize this as some kind of hallucination+streisand effect combo...

I would call it citogenesis or circular reporting. Or perhaps machine citogenesis or model citogenesis.

https://xkcd.com/978/

https://en.wikipedia.org/wiki/Circular_reporting

Re: A small number of samples can poison LLMs of any size

#238

Post by bix6 » Fri, Oct 10, 2025, 2:18 AM UTC

dotancohen wrote: post #186

whatevertrevor wrote: post #91

Earlier quoted context omitted.

As a user I'm worried about a + b sure. As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs? Is it possible to clean the model on the fly by identifying and removing the poisoning sources post training? Or do you have to start from scratch?

> As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs? As an AI company, why are you training on documents that you haven't verified? The fact that you present your argument as a valid concern is a worrying tell for your entire industry.

AI companies gave up on verification years ago. It’s impossible to verify such intense scraping.

Re: A small number of samples can poison LLMs of any size

#239

Post by cowboylowrez » Fri, Oct 10, 2025, 2:19 AM UTC

habinero wrote: post #198

criemen wrote: post #106

Earlier quoted context omitted.

> Please provide a citation for wild claims like this. Even "reasoning" models are not actually reasoning, they just use generation to pre-fill the context window with information that is sometimes useful to the task, which sometimes improves results. That seems to be splitting hairs - the currently-accepted industry-wide definition of "reasoning" models is that they use more test-time compute than previous model gen…

> currently-accepted industry-wide definition of "reasoning" You can't both (1) declare "reasoning" to be something wildly different than what humans mean by reasoning and (2) insist people are wrong when they use the normal definition say models don't reason. You gotta pick a lane.

I don't think its too problematic, its hard to say something is "reasoning" without saying what that something is, for another example of terms that adjust their meaning to context for example, the word "cache" in "processor cache", we know what that is because its in the context of a processor, then there's "cache me outside", which comes from some tv episode.

Re: A small number of samples can poison LLMs of any size

#240

Post by whatevertrevor » Fri, Oct 10, 2025, 2:19 AM UTC

dotancohen wrote: post #186

whatevertrevor wrote: post #91

Earlier quoted context omitted.

As a user I'm worried about a + b sure. As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs? Is it possible to clean the model on the fly by identifying and removing the poisoning sources post training? Or do you have to start from scratch?

> As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs? As an AI company, why are you training on documents that you haven't verified? The fact that you present your argument as a valid concern is a worrying tell for your entire industry.

> As an AI company, why are you training on documents that you haven't verified?

Because "I" need to constantly ship out the next iteration of hotness because AGI is around the corner? Because "I" don't know how to verify documents for poison text in a scalable manner? Because "I" don't care? I am not an AI company, how would I know?

For clarity: I'm using "As an AI company" just to indicate the shift in perspective when it comes to defending attack vectors. Not literally indicating that I am (or affiliated with) an AI company.

I am currently happily retired, and planning to stay that way assuming the AI bubble crash doesn't take my retirement egg with it, in a wider market crash. I have no horse in this race, I haven't been convinced by many AI acceleration stories (though admittedly I haven't given the tools a proper shot because for hobby projects I like to do things myself). And it's definitely not my (entire) industry. So completely wrong read on many levels there, friend.