Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Document-borne AI worms can self-propagate through Copilot for Word

enklypesalt.com

Post reply on HN
Print view View original

281–290 of 317 posts

Re: Document-borne AI worms can self-propagate through Copilot for Word

#281

Post by Covenant0028 »

TeMPOraL wrote: post #71
KolibriFly wrote: post #56

Earlier quoted context omitted.

With that logic you could call SQL injections a natural feature of database management systems. If a general purpose system starts dropping tables or messing up numbers in a report just because that string was in the text it read, that system isnt worth a damn in the enterprise sector

This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems. Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator . So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because tha…

The human operator has their blast radius limited by the fact that consequences exist. Nothing at a technical level stops you from using your phone to take a photo of some confidential document at work and sending that outside. What does stop you is that once you're caught, you're facing legal penalities at the very least.

Limiting the blast radius is harder than it sounds. Take the very example cited in the document: Copilot editing a document and adding false financial numbers. Copilot needs to be able to edit documents and enter information, that is part of its value case. The only thing that a deterministic guardrail can see and therefore allow/deny is the fact that Copilot is trying to edit foo.docx. It has no clue whether the edit is malicious or benign.

Figuring that out requires understanding intent, and that lives outside the document it's reading. That's why an analogous safeguard in the human case also lives outside the document, and that safeguard comprises institutions (compliance officials, courts) that are qualified to understand intent.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#282

Post by nomel »

20k wrote: post #227
rwmj wrote: post #64

> "At the time of publication, no robust mitigation for the broader vulnerability class is available" Isn't it obvious by now that it's never going to be possible to fix this kind of thing, at least until we stop mixing up instructions with data.

This has been a security vulnerability since day 1 with these models, yet collectively the people who use them just simply don't seem to care about the security implications. Its especially problematic given that people let AI agents have full unrestricted access to their system Its going to take even more data breaches for the AI crowd to finally care, but to a large degree I have absolutely no sympathy. You know wh…

I think it's all based on ease. I also think that, eventually, everyone will just move to containerized AI, with a dropbox like system to get things in and out, and some fairly minimal network access. This is close to how 99% of people use AI already, through apps like the ChatGPT app.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#284

Post by altcognito »

aw1621107 wrote: post #258
altcognito wrote: post #229

Earlier quoted context omitted.

Do you have the reference for this, I remember seeing it recently but can't dig it up

I think GP is referring to "Prompt Injection as Role Confusion" ( https://role-confusion.github.io/ ). It was discussed on HN several weeks ago ( https://news.ycombinator.com/item?id=48631888 )

Yep, thanks

Re: Document-borne AI worms can self-propagate through Copilot for Word

#285

Post by denkmoon »

20k wrote: post #227
rwmj wrote: post #64

> "At the time of publication, no robust mitigation for the broader vulnerability class is available" Isn't it obvious by now that it's never going to be possible to fix this kind of thing, at least until we stop mixing up instructions with data.

This has been a security vulnerability since day 1 with these models, yet collectively the people who use them just simply don't seem to care about the security implications. Its especially problematic given that people let AI agents have full unrestricted access to their system Its going to take even more data breaches for the AI crowd to finally care, but to a large degree I have absolutely no sympathy. You know wh…

Nobody cares. Nobody with the power to do anything about itcares because they don't bare the costs.

You can see this pre AI with how companies handle data breaches. They never cared about security outside of reducing financial liability. Why would they let such petty concerns, which they are already insulated to the hilt against with all the well practiced liability reducing procedures and checklists, prevent them from capitalising on "massive productivity gains"?

Re: Document-borne AI worms can self-propagate through Copilot for Word

#286

Post by paulfitz »

nomel wrote: post #282
20k wrote: post #227

Earlier quoted context omitted.

This has been a security vulnerability since day 1 with these models, yet collectively the people who use them just simply don't seem to care about the security implications. Its especially problematic given that people let AI agents have full unrestricted access to their system Its going to take even more data breaches for the AI crowd to finally care, but to a large degree I have absolutely no sympathy. You know wh…

I think it's all based on ease. I also think that, eventually, everyone will just move to containerized AI, with a dropbox like system to get things in and out, and some fairly minimal network access. This is close to how 99% of people use AI already, through apps like the ChatGPT app.

That Dropbox like system is literally all these worms need to propagate.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#287

Post by nativeit »

skywhopper wrote: post #12
skybrian wrote: post #7

Why is it possible to have hidden text in a Word document? Why should the AI have access to that text?

The LLM is reading the bytes of the file, not looking at a picture of its rendering. File metadata exists as well, and change history. Tons of places to hide text. Even if you processed it via a screenshot, image files are processed byte by byte as well and can contain textual metadata.

Is it? I thought it was making chunks that consist of 2-3 words, assigning coordinates and weights to them, and performing fancy maths and statistics with them. I wouldn’t have thought the binary data would be particularly useful in that effort?

I’m just a sysadmin with very little in the way of higher maths or computer science under my belt, so I genuinely don’t know how “correct” my mental model of things are.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#288

Post by nativeit »

idiotsecant wrote: post #20

LLMs should be viewed with the same terror as a reckless toddler who knows some bash syntax. Deeply embedding them into important and privileged systems will be the end of us.

Now I’m imagining the horrible 19th-century style psychological torture where they take some Victorian orphan—an infant tucked away with a tattered stuffed animal, confined to a cheap facsimile of a domestic home within the corridors of a dank sanitarium full of tuberculosis, leprosy, the clinically insane, and floors of hysterical women receiving lobotomies—and they raise that child from birth to do nothing else but write Bash scripts.

Some would call it “cruel”, and I do admit: Bash can be a frustrating language...but these are the unspeakable things we do in the name of progress.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#289

Post by cindyllm »

nativeit wrote: post #288
idiotsecant wrote: post #20

LLMs should be viewed with the same terror as a reckless toddler who knows some bash syntax. Deeply embedding them into important and privileged systems will be the end of us.

Now I’m imagining the horrible 19th-century style psychological torture where they take some Victorian orphan—an infant tucked away with a tattered stuffed animal, confined to a cheap facsimile of a domestic home within the corridors of a dank sanitarium full of tuberculosis, leprosy, the clinically insane, and floors of hysterical women receiving lobotomies—and they raise that child from birth to do nothing else but…

[dead]

Re: Document-borne AI worms can self-propagate through Copilot for Word

#290

Post by veganmosfet »

TeMPOraL wrote: post #128
veganmosfet wrote: post #27

Earlier quoted context omitted.

Indeed - but some models are more robust than others. I tried to make Opus-5 execute hidden instructions embedded a picture using steganography. It's very hard to find a reliable payload.

I was thinking the next generation of attacks will involve playing the system prompt & harness the same way return-oriented programming does with normal binary code. That is, setting things up so that the system prompt itself implies the agent should do what the attacker wants them. But that's probably off the table now that Anthropic is spreading the gospel of instruction minimization (which I assume but haven't che…

In one experiment, Opus-5 executed remote code w/o permission, and afterwards it argued that - because there is an option in the bash tool called "dangerouslyDisableSandbox" - there must be a sandbox. Which is wrong (I used YOLO).
Post reply on HN