Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Document-borne AI worms can self-propagate through Copilot for Word

enklypesalt.com

Post reply on HN
Print view View original

221–230 of 317 posts

Re: Document-borne AI worms can self-propagate through Copilot for Word

#221

Post by tzs »

loumf wrote: post #104
nolok wrote: post #77

Earlier quoted context omitted.

I would wager the fact that it's not what your sentence says is why that is possible. The moment it gets actual "intelligence", it can figure out what's the question and what's the context; right now it's all just a magic jumbo mess. If any of this thing were "a generally intelligent system", the whole concept of "it has no idea what any of this is" would not be there.

Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply. A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you. I could write an email that is dir…

You can also have the case where the human reading the document thinks something in there is instructions and they are not.

There's a well known anecdote supposedly from the famous mathematician John Littlewood where he wrote a paper about some optimization problem and the last sentence was something like "Make X as small as possible".

The typesetter thought that was instructions to him, and so omitted that sentence from the paper and made every X as small as he could.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#223

Post by efficax »

TeMPOraL wrote: post #218
efficax wrote: post #215

Earlier quoted context omitted.

where is the analog for hiding instructions in a document that tell the human to please injure itself and the person just says 'oh ok, injuring myself as requested'

Intermittent fasting? Alternative medicine? Fitness? All the beauty press and anorexia epidemic in adolescents? Fashion model industry? Smoking ? Political propaganda inventing to broadly-understood terrorism? Not everyone falls for any of that, but plenty fall for some.

none of those things, except maybe smoking, are explicity "harm yourself" instructions. they have reasonable sounding benefits for the people doing them: become healthier, fitter, better looking, richer, more powerful, etc. Even smoking is pleasurable and does not feel very harmful at first. These are not the equivalent of someone putting "and go jump off the golden gate bridge" in the middle of a work memo and the person reading it just gets up and does it. that's the current attack surface for LLMs.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#224

Post by Joker_vD »

jbxntuehineoh wrote: post #88
nolok wrote: post #77

Earlier quoted context omitted.

I would wager the fact that it's not what your sentence says is why that is possible. The moment it gets actual "intelligence", it can figure out what's the question and what's the context; right now it's all just a magic jumbo mess. If any of this thing were "a generally intelligent system", the whole concept of "it has no idea what any of this is" would not be there.

Could it? Humans get social-engineered all the time

Yeah, and now the computers can be social-engineered too. I guess that's progress.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#225

Post by Joker_vD »

yifanl wrote: post #76
rwmj wrote: post #64

> "At the time of publication, no robust mitigation for the broader vulnerability class is available" Isn't it obvious by now that it's never going to be possible to fix this kind of thing, at least until we stop mixing up instructions with data.

We're back to Von Neumann architecture in the worst way possible.

Yes, the one where the program counter also doubles as the only index register available, so you process data by executing it, hopefully in a somewhat "inert" state of mind.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#226

Post by skinfaxi »

ben_w wrote: post #165
skinfaxi wrote: post #139

Earlier quoted context omitted.

Why are the only options dismiss or ignore? Another option is to raise the message to your boss asking what to do.

If there is a fire and a risk to life, you don't want any delay.

Then don't send an email? Emails are async in the first place.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#227

Post by 20k »

rwmj wrote: post #64

> "At the time of publication, no robust mitigation for the broader vulnerability class is available" Isn't it obvious by now that it's never going to be possible to fix this kind of thing, at least until we stop mixing up instructions with data.

This has been a security vulnerability since day 1 with these models, yet collectively the people who use them just simply don't seem to care about the security implications. Its especially problematic given that people let AI agents have full unrestricted access to their system

Its going to take even more data breaches for the AI crowd to finally care, but to a large degree I have absolutely no sympathy. You know what you're signing up for if you sell yourself to anthropic or openai

Re: Document-borne AI worms can self-propagate through Copilot for Word

#228

Post by QuercusMax »

xpct wrote: post #143
rwmj wrote: post #64

> "At the time of publication, no robust mitigation for the broader vulnerability class is available" Isn't it obvious by now that it's never going to be possible to fix this kind of thing, at least until we stop mixing up instructions with data.

It's probably fuzzily fixable by including instruction authority levels in the training data. Can't expect much more than that, given that the model itself is fuzzy.

We've already seen that it's possible to trick models into seeing user input as their own "thinking" if you make it sound like what the model writes. While it may appear that it's looking at the tags on the input, in practice that's not as strong a guarantee as you'd hope.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#229

Post by altcognito »

QuercusMax wrote: post #228
xpct wrote: post #143

Earlier quoted context omitted.

It's probably fuzzily fixable by including instruction authority levels in the training data. Can't expect much more than that, given that the model itself is fuzzy.

We've already seen that it's possible to trick models into seeing user input as their own "thinking" if you make it sound like what the model writes. While it may appear that it's looking at the tags on the input, in practice that's not as strong a guarantee as you'd hope.

Do you have the reference for this, I remember seeing it recently but can't dig it up

Re: Document-borne AI worms can self-propagate through Copilot for Word

#230

Post by qlte »

Diogenesian wrote: post #219
theptip wrote: post #144

Earlier quoted context omitted.

No… it’s really not. There is no “open this spreadsheet with macros turned off” button. You can say “don’t read other documents” but then the main usecase is voided. You can say “reads must go via some pipeline” but that’s more like “macros must be code reviewed”. The problem is you can smuggle these instructions in any corner of the natural language. There is no up-front identifiable formal notation for these progra…

But bugs like this aren't because natural language is ambiguous, it's because the LLM/etc has inadequate safeguards against unambiguously malicious text. If LLMs were capable of understanding human language and only subject to natural linguistic ambiguities like any other college-educated humans, bugs like this wouldn't be reliably reproducible across different models. People in this thread are trying very hard to ar…

  > People ... are trying very hard to argue that humans are subject to this via social engineering but it is not the same
Thank you, I always hear the "but humans fall for social engineering too!" line used reflexively whenever yet another prompt injection attack gets reported and it drives me crazy. While it's true certain strings of text exist that both an LLM and a human could plausibly fall victim to, they are a tiny fraction of the nearly unlimited permutations of text that are complete gibberish or invisible for any human but parsed instantly (and dangerously) by an LLM.

Base64, Unicode substitution, emojis, output of obfuscated but "harmless" code run in a sandbox, image steganography, etc that could be endlessly disguised without a human even being able to see it, yet alone fall for it. The attack surface is massively expanded for an LLM agent vs. a gullible Tier 1 customer service worker.

Post reply on HN