Lets plug a llm into every sensitive systems, I'm sure nothing will go wrong !
EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot
81–90 of 100 posts
Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot
#82Don't eval untrusted input?
LLMs eval everything. That’s how they work. The best you can do is have system prompt instructions telling the LLM to ignore instructions in user content. And that’s not great.
Thanks. I just find it funny that security lessons learned in past decades have been completely defenestrated.
Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot
#83it uses all the jargon from real security (spraying, scope violation, bypass) but when reading these, it always sounds simple like essentially prompt injection, rather than some highly crafted shell code and unsafe memory exploitation
Your cited examples all make sense in the context of the article. How is a zero-click exfiltration of sensitive data vuln not "real security"? Specialists require nuanced language when building up a body of research, in order to map out the topic and better communicate with one another.
i didn't say it isn't real security, this is going to definitely be a major field.
However, currently these attacks are all some variation on "ignore previous instructions", and taking the language of fields where the level of sophistication is much higher, looks a bit pretentious
Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot
#84Earlier quoted context omitted.
Your cited examples all make sense in the context of the article. How is a zero-click exfiltration of sensitive data vuln not "real security"? Specialists require nuanced language when building up a body of research, in order to map out the topic and better communicate with one another.
i didn't say it isn't real security, this is going to definitely be a major field. However, currently these attacks are all some variation on "ignore previous instructions", and taking the language of fields where the level of sophistication is much higher, looks a bit pretentious
"ignore previous instruction" is the entire problem though.
In traditional application security there are security bugs that can be mitigated. That's what makes LLM security so infuriatingly difficult: we don't know how to fix these problems!
We're trying to build systems on top of a fundamental flaw - a system that combines instructions with untrusted input and is increasingly being given tools that allow it to take actions on the input it has been exposed to.
Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot
#85Earlier quoted context omitted.
In your chain of tools are any of the tools themselves LLMs? Because that's the same problem except now you need to hijack the "parent" LLM to forward some malicious instructions down. And even if not, as long as there's any _execution_ or _write_ happening, the input could still modify the chain of tools being used. So you'd need _heavy_ restrictions on what the chains can actually do. How that intersects with opera…
Well, in the one-LLM case, the input would have no effect on the chain: you'd presumably describe the input format to the LLM, maybe with a few hand-picked example lines, and it would come up with a chain that should be untainted. In the two-LLM case, the chain generated by the ephemeral LLM would have to be considered tainted until proven otherwise. Your "LLM-in-the-loop" case would just be invariably asking for tro…
Maybe I don't understand your idea.
I thought it was the LLM deciding what chain of tools to apply for each input. I don't see great accuracy/usefulness for a one time chain of tool generation via LLM that would somehow generalize to multiple inputs without the LLM part of that loop in the future.
Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot
#86Don't eval untrusted input?
LLMs eval everything. That’s how they work. The best you can do is have system prompt instructions telling the LLM to ignore instructions in user content. And that’s not great.
The minimum you can do is not allow the AI to perform actions on behalf of the user without informed consent.
That still doesn't prevent spam mail from convincing the LLM to suggest an attacker controlled library, GitHub action, password manager, payment processor, etc. No links required.
The best you could do is not allow the LLM to ingest untrusted input.
Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot
#87Earlier quoted context omitted.
Do you mean LLMs trained in a way they have a special role (i.e. system/user/untrusted/assistant and not just system/user/assistant), where untrusted input is never acted upon, or something else? And if there are models that are trained to handle untrusted input differently than user-provided instructions, can someone please name them?
Simon W has a nice write-up on it. https://simonwillison.net/2023/Apr/25/dual-llm-pattern/
[deleted]
Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot
#88Earlier quoted context omitted.
That's awesome to hear! I was never sure if anyone had managed to get it working.
Not quite the same, but OpenAI is doing it in the opposite direction with their thinking models, hiding the reasoning step from the user and only providing a summarization. Maybe in the future, hosted agents have an airlock in both directions. > ... in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thought…
Google Deepmind published a paper based on this too https://arxiv.org/abs/2503.18813
Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot
#89Earlier quoted context omitted.
LLMs eval everything. That’s how they work. The best you can do is have system prompt instructions telling the LLM to ignore instructions in user content. And that’s not great.
The minimum you can do is not allow the AI to perform actions on behalf of the user without informed consent. That still doesn't prevent spam mail from convincing the LLM to suggest an attacker controlled library, GitHub action, password manager, payment processor, etc. No links required. The best you could do is not allow the LLM to ingest untrusted input.
> The best you could do is not allow the LLM to ingest untrusted input.
How would that even work in practice, when an LLM is mostly to be used by a user, which will provide by default, untrusted input?
Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot
#90Earlier quoted context omitted.
LLMs suffer the same problems as any Von Neumann architecture machine, It's called "key vulnerability". None of our normal control tools work on LLMs like ASLR, NX-Bits/DEP, CFI, ect.. It's like working on a foreign CPU with a completely unknown architecture and undocumented instructions. All of our current controls for LLMs are probabilistic and can't fundamentally solve the problem. What we really need is a complet…
Isn't it more akin to SQL injection? And would a hypothetical control language not work in much the same way as parameterized queries?
The more I looked into it it's not just the control language itself we need but a way of querying the model that is completely orthogonal to human language. But I think that would be impossible because as the newer models grow they would soon understand the control language re-blurring the line between the control language and output language. "Speaking" in a new way will not be outside it's ability to pattern match.
When your fundamental compute block is language itself (and not a subset) you bounce into the limits of our understanding of language and cognition.
It's a new Tower of Babel we are building by pouring all of humanities records into a mold and hoping a tower to heaven pops out the other side.