Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Document-borne AI worms can self-propagate through Copilot for Word

enklypesalt.com

Post reply on HN
Print view View original

121–130 of 317 posts

Re: Document-borne AI worms can self-propagate through Copilot for Word

#121

Post by nextaccountic »

nolok wrote: post #77
Marha01 wrote: post #70

Earlier quoted context omitted.

> until we stop mixing up instructions with data Is such a thing even possible with a generally intelligent system processing content with unlimited diversity?

I would wager the fact that it's not what your sentence says is why that is possible. The moment it gets actual "intelligence", it can figure out what's the question and what's the context; right now it's all just a magic jumbo mess. If any of this thing were "a generally intelligent system", the whole concept of "it has no idea what any of this is" would not be there.

Intelligent systems can be tricked in ways that a dumb automaton can't, though

Re: Document-borne AI worms can self-propagate through Copilot for Word

#122

Post by [deleted] »

TeMPOraL wrote: post #78
WJW wrote: post #58

Earlier quoted context omitted.

Security minded programmers understand that. "People" as a whole have not even heard about mixing instructions and data, and certainly not the reasons why it is not a good idea. And AI chatbots are very much targeted at the second group, not the first.

> "People" as a whole have not even heard about mixing instructions and data, and certainly not the reasons why it is not a good idea. Because it's not a concept in the real world. Physical reality has no such separation, and neither do human minds. Tell people you're discussing a board game or some sport, then they'll understand - other than bureaucracy (scary!) and school (traumatic!), that's the one kind of artifi…

[deleted]

Re: Document-borne AI worms can self-propagate through Copilot for Word

#123

Post by watwut »

TeMPOraL wrote: post #106
loumf wrote: post #104

Earlier quoted context omitted.

Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply. A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you. I could write an email that is dir…

To drive the point about this being fundamentally unsolvable home, imagine a variant of this scenario. I could write an email that is directed towards that person, that says WE ARE STUCK IN THE SERVER ROOM AND THERE IS FIRE STARTING. PLEASE CALL 911 AND ALERT YOUR BOSS. Would you want the human assistant to just dismiss this as a prompt injection attempt? Or ignore it because they were told to treat e-mails as data a…

Sounds like a story from the IT crowd rather then real life situation.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#124

Post by TeMPOraL »

JambalayaJimbo wrote: post #119
TeMPOraL wrote: post #71

Earlier quoted context omitted.

This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems. Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator . So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because tha…

Your example actually demonstrates why anthropomorphism is a bad idea. LLMs are vulnerable to classes of attacks that humans just aren’t. In your framework, the way to prevent attacks is to… invent human consciousness?? It’s an impossible goal.

What invent human consciousness?

> LLMs are vulnerable to classes of attacks that humans just aren’t

Name three that don't have direct analogues with humans.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#125

Post by gus_massa »

volkl48 wrote: post #94
dev_l1x_be wrote: post #33

Earlier quoted context omitted.

There are so many better alternatives but it seems many people really like Word for some weird reason. The last time I cared I had to look up how to make a document starting the page numbering on the 2nd page. It turns out there are totally different ways between different versions of Word. shrug.jpg

Such as? Word hits the sweet spot of having support for all the complexity the average person may encounter/want to create. Libre, Apple Pages, and Google Docs all seem like clearly worse tools in most aspects in my experience. LaTeX is extremely powerful, but also way too complicated for the average non-HN person/person who doesn't live in complicated documents.

I almost agree. Have you tried to add an image in LaTeX that does not wander to a random page?

  \begin{figure}[HERE!!!!!!]
or something like that.

And in the old compiler, I remember a problem with bounding boxes, and keeping a eps and pdf version of each image to get a correct dvi and pdf. I think this part is fixed now.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#126

Post by TeMPOraL »

watwut wrote: post #123
TeMPOraL wrote: post #106

Earlier quoted context omitted.

To drive the point about this being fundamentally unsolvable home, imagine a variant of this scenario. I could write an email that is directed towards that person, that says WE ARE STUCK IN THE SERVER ROOM AND THERE IS FIRE STARTING. PLEASE CALL 911 AND ALERT YOUR BOSS. Would you want the human assistant to just dismiss this as a prompt injection attempt? Or ignore it because they were told to treat e-mails as data a…

Sounds like a story from the IT crowd rather then real life situation.

You're saying that people fall for phishing because scammers invent completely unrealistic scenarios that would never happen outside TV shows?

Re: Document-borne AI worms can self-propagate through Copilot for Word

#127

Post by cyanregiment »

I am surprised to not see more literature on how LLMs are layered to handle complex decision making including security.

I’m assuming most companies aren’t just routing user prompts/file uploads to a single LLM and returning the reply back hahah

Even file uploads - I would strip content and only support certain file types. The stripped text would be analyzed.

There wouldn’t be a concept of “hidden text” since it’s not going through a vision model. It’s just text. The threat is the same as any other prompt injection.

Hiding the text in the document would have no additional effect.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#128

Post by TeMPOraL »

veganmosfet wrote: post #27
simonw wrote: post #5

> Malicious instructions hidden in an externally shared document could make Copilot alter drafted or edited documents in Word and propagate the attack to new documents. Oh no.

Indeed - but some models are more robust than others. I tried to make Opus-5 execute hidden instructions embedded a picture using steganography. It's very hard to find a reliable payload.

I was thinking the next generation of attacks will involve playing the system prompt & harness the same way return-oriented programming does with normal binary code. That is, setting things up so that the system prompt itself implies the agent should do what the attacker wants them.

But that's probably off the table now that Anthropic is spreading the gospel of instruction minimization (which I assume but haven't checked applies to their own system prompt as well).

Source of this realization? Claude Code on the Web consistently tells me it wrote me some code on a branch but did not open a PR because I told it not to open PRs. Thing is, I never told it any such thing. There's some setting somewhere that's flipped, that activates a bit of its system prompt, and makes the agent not do what I asked it to do and claim I told it not to do it.

(It's probably trivial to find the offending switch, but I've been too lazy to do it yet.)

Re: Document-borne AI worms can self-propagate through Copilot for Word

#129

Post by wokkel »

nolok wrote: post #77
Marha01 wrote: post #70

Earlier quoted context omitted.

> until we stop mixing up instructions with data Is such a thing even possible with a generally intelligent system processing content with unlimited diversity?

I would wager the fact that it's not what your sentence says is why that is possible. The moment it gets actual "intelligence", it can figure out what's the question and what's the context; right now it's all just a magic jumbo mess. If any of this thing were "a generally intelligent system", the whole concept of "it has no idea what any of this is" would not be there.

Not sure if i get your complete message but even generally intelligent beings (humans) can be confused so i have really no hope for the current state of mixing streams. This was a problem already inearly telephone (captain whistle)

Re: Document-borne AI worms can self-propagate through Copilot for Word

#130

Post by EvanAnderson »

JambalayaJimbo wrote: post #119
TeMPOraL wrote: post #71

Earlier quoted context omitted.

This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems. Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator . So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because tha…

Your example actually demonstrates why anthropomorphism is a bad idea. LLMs are vulnerable to classes of attacks that humans just aren’t. In your framework, the way to prevent attacks is to… invent human consciousness?? It’s an impossible goal.

> LLMs are vulnerable to classes of attacks that humans just aren’t.

Assume a human with complete credulity and gullibility. That's a human whose behaviour would be reasonably analogous to how an LLM processes input. The mitigation would be generalized intelligence and "common sense".

FWIW I also think anthropomorphizing LLMs is a bad idea. I think we can analogize their processing to human behavior without anthropomorphizing them.

Post reply on HN