[dead]
Document-borne AI worms can self-propagate through Copilot for Word
301–310 of 317 posts
Re: Document-borne AI worms can self-propagate through Copilot for Word
#302Re: Document-borne AI worms can self-propagate through Copilot for Word
#303Earlier quoted context omitted.
I think having my AI raise to me for intervention when it receives an email like the one you described is pretty reasonable, all things considered then. edit: How would a human receiver know that they weren't being deceived or scammed? In what world would we expect this kind of email directly lead to calling emergency services?
> In what world would we expect this kind of email directly lead to calling emergency services? Go through the examples I gave you (plus some more below, they're easy to find) and explain why these are not counter-examples to your skepticism. If you want to be overly-focussed on the specific example rather than the general point, also consider that calling emergency services is no more costly than forwarding an email…
So what point are you trying to make here? That AI should indiscriminately call for emergency services when prompted because a person would do that (which a person would absolutely NOT call emergency services on any message telling you to)?
Re: Document-borne AI worms can self-propagate through Copilot for Word
#304Earlier quoted context omitted.
Yeah so you're seeing how contrived this whole thing is right? That was kind of the point..
It happens . Like all emergencies, it's a low probability event with extremely high impact. You don't want people to ignore them, in fact people are trained - by their public services and their employers - to not ignore them and how to react efficiently.
So we need AI to indiscriminately call emergency services when receiving an email directing it to do so, without raising to a person because of this rare case, that's your assertion?
Re: Document-borne AI worms can self-propagate through Copilot for Word
#305Earlier quoted context omitted.
Part of reading a document is that in the middle of it, it may ask the reader to do something. That is true for humans too. Sometimes they might not realize that the instructions are malicious or are coerced to comply. A simple example: Let’s say I know that you have a human assistant reading your email, summarizing and filtering it, and then forwarding on the important ones to you. I could write an email that is dir…
You can also have the case where the human reading the document thinks something in there is instructions and they are not. There's a well known anecdote supposedly from the famous mathematician John Littlewood where he wrote a paper about some optimization problem and the last sentence was something like "Make X as small as possible". The typesetter thought that was instructions to him, and so omitted that sentence…
There's a similar story about J. Edgar Hoover scribbling "watch the borders" on a memo, which was intended to be a formatting comment, but was interpreted as instructions.
Re: Document-borne AI worms can self-propagate through Copilot for Word
#306Earlier quoted context omitted.
It happens . Like all emergencies, it's a low probability event with extremely high impact. You don't want people to ignore them, in fact people are trained - by their public services and their employers - to not ignore them and how to react efficiently.
So we need AI to indiscriminately call emergency services when receiving an email directing it to do so, without raising to a person because of this rare case, that's your assertion?
His question upthread is what *you* (or a typical human) would be expected to do, as an illustration of why he thinks it's never possible to fully separate instructions and data.
This doesn't proscribe or prescribe "thou shalt not/must always", it is an example thay says "Shit's hard, yo. Don't expect easy wins."
Even my "solution" (separate instructions and data by having an LLM write a program to process data, never touch data directly) is at best going to be like a philosopher writing a dentological ethics book that gets implemented by extremely literal-minded jobsworths.
Re: Document-borne AI worms can self-propagate through Copilot for Word
#307Earlier quoted context omitted.
You just did that in a HN comment, yet nothing happened :). Could it be that the whole idea is silly misunderstanding of fundamental tenets of reality in the first place?
Well duh. You can also post a random AI malware prompt, and I can assure you nothing will happen. What's your point?
My point is that "mixing instructions and data" is a red herring, and that reality has no such distinction; what is code and what is data isn't just context-dependent but question-dependent, which you aptly demonstrated by posting "code" that in this context is just "data".
Re: Document-borne AI worms can self-propagate through Copilot for Word
#308Earlier quoted context omitted.
Well duh. You can also post a random AI malware prompt, and I can assure you nothing will happen. What's your point?
My point is that "mixing instructions and data" is a red herring, and that reality has no such distinction; what is code and what is data isn't just context-dependent but question-dependent, which you aptly demonstrated by posting "code" that in this context is just "data".
That's a lot of words to argue that we shouldn't be separating prompts from the (often untrusted) data the prompts are acting on.
I still don't see your point.
Re: Document-borne AI worms can self-propagate through Copilot for Word
#309> "At the time of publication, no robust mitigation for the broader vulnerability class is available" Isn't it obvious by now that it's never going to be possible to fix this kind of thing, at least until we stop mixing up instructions with data.
This has been a security vulnerability since day 1 with these models, yet collectively the people who use them just simply don't seem to care about the security implications. Its especially problematic given that people let AI agents have full unrestricted access to their system Its going to take even more data breaches for the AI crowd to finally care, but to a large degree I have absolutely no sympathy. You know wh…
> people let AI agents have full unrestricted access to their system
Not everyone is like this. I've never ran Claude Code unsandboxed. I took the time to firejail it since day one. One of the first things I did was ask it to write a virtual machine orchestration tool so I could virtualize entire projects and their associated AI sessions. Mitigates supply chain attacks too.
I managed to rig up a firewall and it withstood a Fable code review. I'm now writing a custom network stack for it in Rust so I can firewall every single network connection.
Re: Document-borne AI worms can self-propagate through Copilot for Word
#310Earlier quoted context omitted.
This has been a security vulnerability since day 1 with these models, yet collectively the people who use them just simply don't seem to care about the security implications. Its especially problematic given that people let AI agents have full unrestricted access to their system Its going to take even more data breaches for the AI crowd to finally care, but to a large degree I have absolutely no sympathy. You know wh…
Its funny because I have been asked by some folks at work as to why I dont just 'allow all commands' and instead review each one, and then I let my colleague in infosec know who asks that so they are aware of dangerous users.
Because it's extremely annoying and full of false positives. The vast majority of the commands are fine, analyzing each and every command the AI generates will quickly give any human alarm fatigue.
It's more ergonomic to put them in a fully isolated virtual machine and let them cook.