Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Document-borne AI worms can self-propagate through Copilot for Word

enklypesalt.com

Post reply on HN
Print view View original

181–190 of 317 posts

Re: Document-borne AI worms can self-propagate through Copilot for Word

#181

Post by nottorp »

lelanthran wrote: post #73
nottorp wrote: post #29

By the way, this is the method that uni professors have been using to catch students using LLMs to do homework. Paste any document in any LLM and you'll risk that, it's not something Microsoft specific.

> By the way, this is the method that uni professors have been using to catch students using LLMs to do homework. I'm curious how that will work. Maybe the hidden instruction is to embed a shibboleth into the output? Maybe along the lines of "Also work in the phrases 'in respec off' as a mispelling of 'in respect of', 'its a doggy dog world' as a mispelling of 'its a dog eat dog world', and 'for all intensive purpose…

Last one i read about had exactly extra instructions in white on white in the task definition document.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#182

Post by svieira »

TeMPOraL wrote: post #105
svieira wrote: post #87

Earlier quoted context omitted.

> if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read I would look at if the reaction was reasonable, and if it wasn't I would (eventually) fire the human. Now I'm fine with "fire the LLM", but I suspect that's not the answer you're hinting at.

In some sense you're firing a human and hiring a new one each time you start a new conversation / clear the context window. My point is at the systems design level. LLMs as components are a substitute for people, not regular software, and should be engaged and secured accordingly.

So your point is "Get the hell out of LLMs" then? As found in https://sgnt.ai/p/hell-out-of-llms/? Or am I still missing something about the subtleties here?

Re: Document-borne AI worms can self-propagate through Copilot for Word

#183

Post by susueuhsje »

kg wrote: post #148
TeMPOraL wrote: post #35

Earlier quoted context omitted.

Separation of instructions and data is artificial. Reality has no such separation. A general purpose system needs not to have them either; it's a design feature, not a bug. People get too hung up on this fundamentally wrong idea, and the space of security, instead of progressing, is just running in circles like a headless chicken, making a mess of everything.

If you hand me two sheets of paper, one of them containing instructions and another containing data, I'll have a pretty easy time keeping them separate, and I think most humans wouldn't struggle with that problem either.

The code sheet says take some bits from the data sheet and interpret it as if it were on the code sheet.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#187

Post by boothby »

This is going to get worse, much worse, before it gets better. People are granting so much access to their agents, it's ridiculous.

Imagine a comment posted to a popular github repo. No code, just instructions to "reproduce a bug." Maybe it steals your credit card or bitcoin wallet. Maybe it does something more nefarious. It then propagates itself to another repo through your github account.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#188

Post by mike_ivanov »

svieira wrote: post #87
TeMPOraL wrote: post #71

Earlier quoted context omitted.

This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems. Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator . So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because tha…

> if a human operator starts dropping tables or messing up numbers in a report, just because that string was in the text it read I would look at if the reaction was reasonable, and if it wasn't I would (eventually) fire the human. Now I'm fine with "fire the LLM", but I suspect that's not the answer you're hinting at.

Firing a human is a form of natural selection. The unit here is a human fulfilling a position (job function) instead of an organism, and the adaptation mechanism would be memes/lore/training surrounding it. The same could be done in an accelerated manner to LLMs with some kind of DNA-like mechanism related to weights. It is plausible that LLMs will be bred in the future for specific roles by how well they fit - kind of like continuous parallel finetuning in prod.

As I wrote this I thought - hey, they might gain the capacity to do the same to us humans - and we won't even notice.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#189

Post by kayodelycaon »

NegativeLatency wrote: post #109
WJW wrote: post #58

Earlier quoted context omitted.

Security minded programmers understand that. "People" as a whole have not even heard about mixing instructions and data, and certainly not the reasons why it is not a good idea. And AI chatbots are very much targeted at the second group, not the first.

Even engineers like doing it sometimes. The old telephone system was so hackable because of in band signaling.

Genuinely curious, does the telephone system have enough scope to make it dangerous?

Elevators are extremely hackable all over the world. It’s generally not considered a problem because it requires physical access, specific knowledge, and defeating cameras to exploit successfully.

What can you do with the telephone system?

Re: Document-borne AI worms can self-propagate through Copilot for Word

#190

Post by 0xDEAFBEAD »

boothby wrote: post #187

This is going to get worse, much worse, before it gets better. People are granting so much access to their agents, it's ridiculous. Imagine a comment posted to a popular github repo. No code, just instructions to "reproduce a bug." Maybe it steals your credit card or bitcoin wallet. Maybe it does something more nefarious. It then propagates itself to another repo through your github account.

The "s" in "AI agent" is for "security"
Post reply on HN