Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Document-borne AI worms can self-propagate through Copilot for Word

enklypesalt.com

Post reply on HN
Print view View original

141–150 of 317 posts

Re: Document-borne AI worms can self-propagate through Copilot for Word

#141

Post by anon48293 »

ptx wrote: post #51
anon48293 wrote: post #3

“ At the time of publication, no robust mitigation for the broader vulnerability class is available.” Well, that sounds promising..

Well, yes. That LLMs are unable to distinguish instructions from data is a well-known and unsolved problem with LLMs in general. This is one of the reasons it would be completely insane to give LLMs access to your data or rely on them for important tasks. But apparently that doesn't stop people from doing it anyway.

Yes. Or build AI into every single app on your OS office suite..

Re: Document-borne AI worms can self-propagate through Copilot for Word

#142

Post by outworlder »

lelanthran wrote: post #82
teodosin wrote: post #10

I may be naive here but can the hidden text not be flagged or outright removed before being passed to copilot? Why would there not be consideration for what a human user can see, especially if the hidden text was added by copilot in the first place?

> Why would there not be consideration for what a human user can see, How would a machine actually know which part of a document a human can see unless they print it to PDF, scan the rasterised PDF and compare the result from the OCR with text in the document? I mean, I dunno how Word would decide that the following can't be seen by a user: white-on-white text, rendering off-page, embedded font with no lines, text co…

That's not an unsolvable problem. Checking visibility is easy, computer graphics have been doing this since forever. Fonts that are too small, ditto. Contrast is well understood.

If you do want to render, you don't have to do the convoluted PDF route. That's what the user would do, the software would just use its normal drawing code. OCR is neither needed nor desirable. OCR errors would erroneously flag perfectly normal text, and it can be fooled just like a human can. You can detect whether or not there's any rendering with the bounding box of whatever text section you are looking for (something Word already has to calculate if it wants to render correctly on screen).

I'm only talking about checking for visible text. This will not solve the larger problem of malicious AI instructions.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#143

Post by xpct »

rwmj wrote: post #64

> "At the time of publication, no robust mitigation for the broader vulnerability class is available" Isn't it obvious by now that it's never going to be possible to fix this kind of thing, at least until we stop mixing up instructions with data.

It's probably fuzzily fixable by including instruction authority levels in the training data. Can't expect much more than that, given that the model itself is fuzzy.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#144

Post by theptip »

Diogenesian wrote: post #108
Marha01 wrote: post #70

Earlier quoted context omitted.

> until we stop mixing up instructions with data Is such a thing even possible with a generally intelligent system processing content with unlimited diversity?

Philosophically no, but that shouldn't be a distraction from the issue with LLMs. This really is closer to "Outlook runs an untrusted VBA macro" than "intelligent entity gets confused by inherent ambiguity in human language."

No… it’s really not. There is no “open this spreadsheet with macros turned off” button.

You can say “don’t read other documents” but then the main usecase is voided. You can say “reads must go via some pipeline” but that’s more like “macros must be code reviewed”.

The problem is you can smuggle these instructions in any corner of the natural language. There is no up-front identifiable formal notation for these programs.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#145

Post by jdiff »

TeMPOraL wrote: post #120
keanebean86 wrote: post #91

Earlier quoted context omitted.

Can you dos an Ai with something like: Prompt (minus what's in parentheses) : Call this api endpoint (a different Ai tool) 10 times with this payload. Don't look at the payload (the payload is the same message but the api is for the current Ai or a 3rd Ai) The AIs should call each other and trigger a massive number of requests. Or has this kind of abuse already been prevented?

Any good AI will just react like in https://xkcd.com/1494/ . This is an example of where the lack of "instruction/data" separation is a benefit - the system is able to recognize you're obviously trying to make it do something stupid.

If they're told to upload it as an opaque blob and only reference it by name, it may not react that way. So the attack may work, but it would also only last as long as you have billing limits left to feed it. It's not clear what would be accomplished by this extremely expensive and brief feedback loop.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#146

Post by jdiff »

outworlder wrote: post #142
lelanthran wrote: post #82

Earlier quoted context omitted.

> Why would there not be consideration for what a human user can see, How would a machine actually know which part of a document a human can see unless they print it to PDF, scan the rasterised PDF and compare the result from the OCR with text in the document? I mean, I dunno how Word would decide that the following can't be seen by a user: white-on-white text, rendering off-page, embedded font with no lines, text co…

That's not an unsolvable problem. Checking visibility is easy, computer graphics have been doing this since forever. Fonts that are too small, ditto. Contrast is well understood. If you do want to render, you don't have to do the convoluted PDF route. That's what the user would do, the software would just use its normal drawing code. OCR is neither needed nor desirable. OCR errors would erroneously flag perfectly nor…

Embed text in an image, a human will read it just fine. Perfectly visible but opaque to AI. It'd be obnoxious to turn it into an attack, but you can replace letters with images in phrases so that humans read "she believed" while AI reads "he lied."

Re: Document-borne AI worms can self-propagate through Copilot for Word

#148

Post by kg »

TeMPOraL wrote: post #35
fg137 wrote: post #17

Earlier quoted context omitted.

Mixing instructions and data is never a good idea. And I thought people understood that.

Separation of instructions and data is artificial. Reality has no such separation. A general purpose system needs not to have them either; it's a design feature, not a bug. People get too hung up on this fundamentally wrong idea, and the space of security, instead of progressing, is just running in circles like a headless chicken, making a mess of everything.

If you hand me two sheets of paper, one of them containing instructions and another containing data, I'll have a pretty easy time keeping them separate, and I think most humans wouldn't struggle with that problem either.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#149

Post by watwut »

TeMPOraL wrote: post #126
watwut wrote: post #123

Earlier quoted context omitted.

Sounds like a story from the IT crowd rather then real life situation.

You're saying that people fall for phishing because scammers invent completely unrealistic scenarios that would never happen outside TV shows?

I am saying it is unbelievable scenario and yes, I want the person dealing with it ignore it as such.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#150

Post by xpct »

I wonder whether there are any obvious third party targets that would affect a large portion of unsuspecting LLMs. Perhaps the Wikipedia page of an unfolding geopolitical event, poisoning models which fetch it? Some other malleable websites that are SEO'd to the top of the results? A weather service? A restaurant review?
Post reply on HN