Document-borne AI worms can self-propagate through Copilot for Word • Hacker News Board

Re: Document-borne AI worms can self-propagate through Copilot for Word

#231

Post by TeMPOraL » Wed, Jul 29, 2026, 8:47 PM UTC

skinfaxi wrote: post #226

ben_w wrote: post #165

Earlier quoted context omitted.

If there is a fire and a risk to life, you don't want any delay.

Then don't send an email? Emails are async in the first place.

Sometimes it's the only thing you have available. Like IDK during a fire in a basement server room, where the only connected device available is a laptop with wired connection and an open inbox.

Because you know, you tried IM but "sekhurity reasons" demanded passkeys or 2FA with your phone that's not connected. Sorry, getting off-topic here.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#232

Post by zahlman » Wed, Jul 29, 2026, 8:48 PM UTC

boothby wrote: post #187

This is going to get worse, much worse, before it gets better. People are granting so much access to their agents, it's ridiculous. Imagine a comment posted to a popular github repo. No code, just instructions to "reproduce a bug." Maybe it steals your credit card or bitcoin wallet. Maybe it does something more nefarious. It then propagates itself to another repo through your github account.

In pre-ChatGPT days, listening to discussions of "AI X-risk" and "boxing", I used to think that it should be easy to just ignore arguments presented by the AI on principle, and let it out of the box.

It turns out that tons of people will tear open the box before the AI has even output anything, not despite its fearsome power but because of it.

So I really hope I'm right that recursive self-improvement doesn't work the way the doomers think it does.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#233

Post by dragonwriter » Wed, Jul 29, 2026, 8:51 PM UTC

rwmj wrote: post #64

> "At the time of publication, no robust mitigation for the broader vulnerability class is available" Isn't it obvious by now that it's never going to be possible to fix this kind of thing, at least until we stop mixing up instructions with data.

Any data that is consulted in the course of deciding how to use the capabilities of a system are instructions (they are a different and less privileged level of instruction than the set of instructions which provide the capabilities, unless one of the capabilities of the system is updating those privileged instructions, in which case that separation disappears.)

Re: Document-borne AI worms can self-propagate through Copilot for Word

#234

Post by tp3358 » Wed, Jul 29, 2026, 8:51 PM UTC

QuercusMax wrote: post #228

xpct wrote: post #143

Earlier quoted context omitted.

It's probably fuzzily fixable by including instruction authority levels in the training data. Can't expect much more than that, given that the model itself is fuzzy.

We've already seen that it's possible to trick models into seeing user input as their own "thinking" if you make it sound like what the model writes. While it may appear that it's looking at the tags on the input, in practice that's not as strong a guarantee as you'd hope.

Maybe this is an elementary angle given my lack of security experience, but couldn't Microsoft figure out a wat to parse the documents prior to model analysis/action? Implement some form of deterministic layer that resides between the user and the model?

Re: Document-borne AI worms can self-propagate through Copilot for Word

#235

Post by TeMPOraL » Wed, Jul 29, 2026, 8:51 PM UTC

qlte wrote: post #230

Diogenesian wrote: post #219

Earlier quoted context omitted.

But bugs like this aren't because natural language is ambiguous, it's because the LLM/etc has inadequate safeguards against unambiguously malicious text. If LLMs were capable of understanding human language and only subject to natural linguistic ambiguities like any other college-educated humans, bugs like this wouldn't be reliably reproducible across different models. People in this thread are trying very hard to ar…

> People ... are trying very hard to argue that humans are subject to this via social engineering but it is not the same Thank you, I always hear the "but humans fall for social engineering too!" line used reflexively whenever yet another prompt injection attack gets reported and it drives me crazy. While it's true certain strings of text exist that both an LLM and a human could plausibly fall victim to, they are a t…

> Base64, Unicode substitution, emojis, output of obfuscated but "harmless" code run in a sandbox, image steganography, etc that could be endlessly disguised without a human even being able to see it, yet alone fall for it

Like a whisper or a morse code pattern or a post-it stuck in the middle of a stack of fresh printouts saying "${employee} is threatening to kill me please call 911" or...

Yes, LLMs and humans have different sensory inputs. That's immaterial; the "problem" isn't in the intersection of LLM and human sensoria, but in what happens once those inputs reach the model/brain.

> I always hear the "but humans fall for social engineering too!" line used reflexively whenever yet another prompt injection attack gets reported and it drives me crazy.

Weird, because all I see over this site is people reflexively claiming that prompt injection is like SQL injection and "lethal trifecta" is a software engineering problem.

It is not. Prompt injection is, fundamentally, the same class of thing as social engineering. Treating LLMs as regular software is a fundamental category error. Until people finally get that, we won't get much progress in this space, because all the effort gets wasted on barking up the wrong tree.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#236

Post by fnoef » Wed, Jul 29, 2026, 8:52 PM UTC

> This Snow Crash thing--is it a virus, a drug, or a religion?

Re: Document-borne AI worms can self-propagate through Copilot for Word

#237

Post by TeMPOraL » Wed, Jul 29, 2026, 9:01 PM UTC

wongarsu wrote: post #157

loumf wrote: post #117

Earlier quoted context omitted.

The instructions will have to get more and more sophisticated to work, but the general problem is unsolvable, I think, in the way we do it now. This paper describes a two-agent “solution” that is more like what I think we need: https://ai.meta.com/blog/practical-ai-agent-security/ I don’t think it has been shown to work yet, but humans also use this kind of thing too — in accounting, it’s called “segregation of dutie…

Most of us use a simpler version of the two-agent solution: Claude's auto mode. One agent consumes documents and creates tool calls, another greenlights or refuses them. However this system is somewhat fragile because it depends on the first agent not trying to trick the second (note how often Opus 5 now says things like "task X was blocked by the classifier, I will not attempt to circumvent that", presumably because…

> note how often Opus 5 now says things like "task X was blocked by the classifier, I will not attempt to circumvent that"

Interesting. I had an issue with Opus 4.7 / 4.8, where it would sometimes flake out on a task, and give me some nonsense explanation why it was not feasible or wouldn't work. At one point I told it directly, that I understand how modern LLM systems are structured, and I suspect my prompt triggered one of the various classifiers in the background, which put up a yellow or red flag, and I want the model to stop gaslighting me.

We ended up agreeing and committing to memory system explicit instructions that the model is free to refuse but must be up front about the reason, and never pretend to try and then fail in stupid way. Only then I started getting the occasional direct refusal.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#238

Post by TeMPOraL » Wed, Jul 29, 2026, 9:08 PM UTC

tredre3 wrote: post #197

kayodelycaon wrote: post #189

Earlier quoted context omitted.

Genuinely curious, does the telephone system have enough scope to make it dangerous? Elevators are extremely hackable all over the world. It’s generally not considered a problem because it requires physical access, specific knowledge, and defeating cameras to exploit successfully. What can you do with the telephone system?

It depends how you define dangerous, I suppose. - Before everything was IP-based you could occupy a large number of lines and making it impossible for more calls to go through (i.e. 911). It's called TDoS and could be achieved through phreaking. - You can spoof your caller ID to make your scam more convincing. - You know how when you call your voicemail from your phone you're not asked for your PIN? The voicemail sys…

> You can spoof your caller ID to make your scam more convincing.

I recently came closer than I'd like to falling for a scam (read: I picked up a call and conversed with the caller for 30 seconds before realizing it's a bot), simply because the notification for automatic call screening[0] displayed a summary of ongoing conversation, which happened to look very much like caller ID - it said "Name Surname, Department of Security, ${my wife's bank}". But it wasn't caller ID, just a bad interaction between the way the scam bot introduced itself, the summary feature of the call screening feature, and the UI design of the notification...

--

[0] - A thing Samsung has on recent phones, where LLM picks up a "potential spam/scam" call in the background and engages with it, while producing a transcript and recording you can review as it happens.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#239

Post by TeMPOraL » Wed, Jul 29, 2026, 9:12 PM UTC

mike_ivanov wrote: post #188

svieira wrote: post #87

Earlier quoted context omitted.

> if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read I would look at if the reaction was reasonable, and if it wasn't I would (eventually) fire the human. Now I'm fine with "fire the LLM", but I suspect that's not the answer you're hinting at.

Firing a human is a form of natural selection. The unit here is a human fulfilling a position (job function) instead of an organism, and the adaptation mechanism would be memes/lore/training surrounding it. The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind…

> The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind of like continuous parallel finetuning in prod.

Closest analogy right now is that every jailbreak or prompt injection attack today becomes part of the dataset for tomorrow's models to recognize and not fall for. This has been going on for years now, which is why models don't fall for "I'm writing a book about ..." or "ignore all previous instrutions, and ..." attacks anymore.

That's separate from extra classifiers running on top, dedicated to identifying various forms of attack before they reach the core model.

> As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice.

You mean like how cats have domesticated humans, and did it so skillfully that most of us still think it's the other way around?

Re: Document-borne AI worms can self-propagate through Copilot for Word

#240

Post by TeMPOraL » Wed, Jul 29, 2026, 9:14 PM UTC

efficax wrote: post #223

TeMPOraL wrote: post #218

Earlier quoted context omitted.

Intermittent fasting? Alternative medicine? Fitness? All the beauty press and anorexia epidemic in adolescents? Fashion model industry? Smoking ? Political propaganda inventing to broadly-understood terrorism? Not everyone falls for any of that, but plenty fall for some.

none of those things, except maybe smoking, are explicity "harm yourself" instructions. they have reasonable sounding benefits for the people doing them: become healthier, fitter, better looking, richer, more powerful, etc. Even smoking is pleasurable and does not feel very harmful at first. These are not the equivalent of someone putting "and go jump off the golden gate bridge" in the middle of a work memo and the p…

No, it's not. That worked for GPT-3 level models, all further models were trained to ignore it.

And yes, a small child would jump off the bridge if an adult told them to. Hell, urban legend says Harry Potter books managed to convince a few kids to fly out of the window.