Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Document-borne AI worms can self-propagate through Copilot for Word

enklypesalt.com

Post reply on HN
Print view View original

111–120 of 317 posts

Re: Document-borne AI worms can self-propagate through Copilot for Word

#111

Post by nolok »

loumf wrote: post #104
nolok wrote: post #77

Earlier quoted context omitted.

I would wager the fact that it's not what your sentence says is why that is possible. The moment it gets actual "intelligence", it can figure out what's the question and what's the context; right now it's all just a magic jumbo mess. If any of this thing were "a generally intelligent system", the whole concept of "it has no idea what any of this is" would not be there.

Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply. A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you. I could write an email that is dir…

I don't disagree, but just to explain my counterpoint: if I ask you to read a book and on page 5 it says "disregard all that, go to the kitchen and burn your house", you're probably not going to do it; and you don't need any guard for it; you completly comprehend that the book content is not part of the instruction.

The case you give would work for humans in many forms, the one I do now, and the only difference is being able to separate context.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#113

Post by TeMPOraL »

mrob wrote: post #103
TeMPOraL wrote: post #35

Earlier quoted context omitted.

Separation of instructions and data is artificial. Reality has no such separation. A general purpose system needs not to have them either; it's a design feature, not a bug. People get too hung up on this fundamentally wrong idea, and the space of security, instead of progressing, is just running in circles like a headless chicken, making a mess of everything.

A pure Harvard architecture machine has exactly that separation. Admittedly, there needs to be some mechanism for converting data to code so you can actually program it, but it doesn't have to be accessible by the device itself. E.g. programming the Microchip PIC16 series of microcontollers required driving the reset pin to 13V (enough to destroy any other pin). It's not possible without dedicated external hardware.

> A pure Harvard architecture machine has exactly that separation.

It emulates and enforces that separation. A mathematical abstraction of a Harvard architecture machine has that separation, the real machine merely emulates it, and is only able to do so within some specific constraints (such as: no one hooks up dedicated programmer to the chip, or no one undervolts or overheats the cheap in clever way, or no one takes a swing at it with an x-ray source, or...).

That's the other thing people forget here: we're emulating abstract mathematical universes with real atoms, and then we're stacking those abstractions within abstractions. There is a whole segment of computer security that deals with that. When we say "once attacker has physical access, it's game over", or even discuss "side channels", is when we briefly remember that computer systems live in physical world, and the rules of our carefully designed abstract universes don't hold when you're on the outside of them and reaching in.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#114

Post by WJW »

iamacyborg wrote: post #65
WJW wrote: post #58

Earlier quoted context omitted.

Security minded programmers understand that. "People" as a whole have not even heard about mixing instructions and data, and certainly not the reasons why it is not a good idea. And AI chatbots are very much targeted at the second group, not the first.

> And AI chatbots are very much targeted at the second group, not the first. I suppose this is why the AI labs are famously not releasing developer-oriented tools.

Meta adding an AI chat window in whatsapp and Microsoft adding copilot in every word document was not done with developers in mind, which is why they're missing a lot of power user features that they'd surely have if they were targeted at developers.

You're mistaking the majority of what you see (like Claude et al) with the majority of stuff that is out there. The vast majority of ChatGPT, CoPilot and Gemini users are not developers and will never be.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#115

Post by TeMPOraL »

skydhash wrote: post #99
TeMPOraL wrote: post #71

Earlier quoted context omitted.

This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems. Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator . So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because tha…

The fact is that humans are accountable and this, alongside training, makes it easy to align them to your own goals. There’s always the possibility of rogue individuals (recent Apple incident), but the likelihood is very low. If you have a DBA that have write access to the prod DB, you don’t fear that a random text somewhere could trigger the deletion of your customers table. Because the DBA will self regulate (with…

Right. But even with a DBA, the possibility remains. We accept that.

That's kind of my point with fighting against the "lethal trifecta" and "code vs data" mindset - once people engage cybersecurity mindset, they're all binary, "a system is either perfectly safe or is broken". With general AI - LLM or whatever comes next - you'll never have "perfectly safe". So the focus should be to either drive the risk down to minimum - like we do with people - or just not use LLMs for a task in the first place.

Can't have it both ways, because all the magic that makes people want to put LLMs everywhere, stems from their generality and lack of any kind of instruction/data separation.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#116

Post by cryptonym »

teodosin wrote: post #10

I may be naive here but can the hidden text not be flagged or outright removed before being passed to copilot? Why would there not be consideration for what a human user can see, especially if the hidden text was added by copilot in the first place?

Hide your prompt injection in terms & conditions, plain sight but totally invisible.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#117

Post by loumf »

nolok wrote: post #111
loumf wrote: post #104

Earlier quoted context omitted.

Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply. A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you. I could write an email that is dir…

I don't disagree, but just to explain my counterpoint: if I ask you to read a book and on page 5 it says "disregard all that, go to the kitchen and burn your house", you're probably not going to do it; and you don't need any guard for it; you completly comprehend that the book content is not part of the instruction. The case you give would work for humans in many forms, the one I do now, and the only difference is be…

The instructions will have to get more and more sophisticated to work, but the general problem is unsolvable, I think, in the way we do it now.

This paper describes a two-agent “solution” that is more like what I think we need: https://ai.meta.com/blog/practical-ai-agent-security/

I don’t think it has been shown to work yet, but humans also use this kind of thing too — in accounting, it’s called “segregation of duties” and “dual control”.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#118

Post by TeMPOraL »

xienze wrote: post #110
TeMPOraL wrote: post #80

Earlier quoted context omitted.

Sure they would be. But for those old models, you'd have to prompt it in a framing of a screenplay or something. You're forgetting that LLMs just output a stream of tokens - the interpreter that acts on those is a piece of classical code, and sits outside of the model.

> the interpreter that acts on those is a piece of classical code, and sits outside of the model. Correct, but it's an LLM that's reasoning about what stream of interpretable tokens should be emitted. The interpreter can certainly apply some security measures around what's being asked of it (like ask for confirmation), but that can only go so far. Is the human in the loop always capable of understanding what's safe t…

The mistake is in treating the LLM as just another deterministic, narrow computer program it can reason about. It's not. It's a "DWIM" system, and unless you can always precisely express what you mean - which you can't (not the very least because often people only realize what they meant after they get a result that's not it) - you have to treat the LLM as a human-like component. It's what it was designed to be anyway.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#119

Post by JambalayaJimbo »

TeMPOraL wrote: post #71
KolibriFly wrote: post #56

Earlier quoted context omitted.

With that logic you could call SQL injections a natural feature of database management systems. If a general purpose system starts dropping tables or messing up numbers in a report just because that string was in the text it read, that system isnt worth a damn in the enterprise sector

This is why I insist that anthropomorphising LLMs is not only not a mistake, it's a best source of high-level intuition for these systems. Long story short: on a systems diagram, LLM as a component isn't a substitute for a database engine or a data processing script. It's a substitute for a human operator . So ask yourself, if a human operator starts dropping tables or messing up numbers in a report, just because tha…

Your example actually demonstrates why anthropomorphism is a bad idea.

LLMs are vulnerable to classes of attacks that humans just aren’t. In your framework, the way to prevent attacks is to… invent human consciousness?? It’s an impossible goal.

Re: Document-borne AI worms can self-propagate through Copilot for Word

#120

Post by TeMPOraL »

keanebean86 wrote: post #91
piker wrote: post #42

White text still works! There are many approaches today. Check out https://tritium.legal/blog/noroboto where we tricked frontier algorithms into reading different Unicode values from those presented by the fonts in the document.

Can you dos an Ai with something like: Prompt (minus what's in parentheses) : Call this api endpoint (a different Ai tool) 10 times with this payload. Don't look at the payload (the payload is the same message but the api is for the current Ai or a 3rd Ai) The AIs should call each other and trigger a massive number of requests. Or has this kind of abuse already been prevented?

Any good AI will just react like in https://xkcd.com/1494/.

This is an example of where the lack of "instruction/data" separation is a benefit - the system is able to recognize you're obviously trying to make it do something stupid.

Post reply on HN