Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

aim.security

Post reply on HN
Print view View original

71–80 of 100 posts

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#71

Post by soulofmischief »

drdaeman wrote: post #64
soulofmischief wrote: post #24

Earlier quoted context omitted.

Double LLM architecture is an increasingly common mitigation technique. But all the same rules of SQL injection still apply: For anything other than RAG, user input should not directly be used to modify or access anything that isn't clientside.

Do you mean LLMs trained in a way they have a special role (i.e. system/user/untrusted/assistant and not just system/user/assistant), where untrusted input is never acted upon, or something else? And if there are models that are trained to handle untrusted input differently than user-provided instructions, can someone please name them?

Simon W has a nice write-up on it. https://simonwillison.net/2023/Apr/25/dual-llm-pattern/

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#72

Post by Emiledel »

simonw wrote: post #48
soulofmischief wrote: post #24

Earlier quoted context omitted.

Double LLM architecture is an increasingly common mitigation technique. But all the same rules of SQL injection still apply: For anything other than RAG, user input should not directly be used to modify or access anything that isn't clientside.

Have you seen that implemented yet?

I've shared a repo here with deterministic, policy driven routing of user inputs so as to operate with it without influencing agent decisions (though it's up to tool calls to take precautions with what they return) https://github.com/its-emile/memory-safe-agent The teams at owasp are great, join us !

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#73

Post by soulofmischief »

simonw wrote: post #48
soulofmischief wrote: post #24

Earlier quoted context omitted.

Double LLM architecture is an increasingly common mitigation technique. But all the same rules of SQL injection still apply: For anything other than RAG, user input should not directly be used to modify or access anything that isn't clientside.

Have you seen that implemented yet?

Oh hey Simon!

I independently landed on the same architecture in a prior startup before you published your dual LLM blog post, though unfortunately there's nothing left standing to show since that company experienced a hostile board takeover, the board squeezed me out of my CTO position in order to plant a yes man, pivoted to something I was against, and then recently shut down after failing to find product-market fit.

I still am interested in the architecture, have continued to play around with it in personal projects, and some other engineers I speak to have mentioned it before, so I think the idea is spreading although I haven't knowingly seen it in a popular product.

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#74

Post by soulofmischief »

Emiledel wrote: post #72
simonw wrote: post #48

Earlier quoted context omitted.

Have you seen that implemented yet?

I've shared a repo here with deterministic, policy driven routing of user inputs so as to operate with it without influencing agent decisions (though it's up to tool calls to take precautions with what they return) https://github.com/its-emile/memory-safe-agent The teams at owasp are great, join us !

I'm very curious how OWASP has been handling LLMs, any good write-ups? What's the best way to get involved?

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#75

Post by Emiledel »

byteknight wrote: post #31
bstsb wrote: post #21

Earlier quoted context omitted.

yeah but i wouldn't really class that as "zero-click" etc. maybe Low interaction required

I have to agree with you. Anything that requires an initiation (a chat in this case) by the user is inherently not "zero-click".

Agree with other comments here - no need for the user to engage with anything from the malicious email, only to continue using their account with some LLM interactions. The account is poisoned even for known safe self initiated interactions.

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#77

Post by LegionMammal978 »

whatevertrevor wrote: post #55
LegionMammal978 wrote: post #39

Earlier quoted context omitted.

> Does the model capable of tool use feed the uninspected user input to a sandboxed model, then treat its output as an opaque string? That was one of my early thoughts for "How could LLM tools ever be made trustworthy for arbitrary data?" The LLM would just come up with a chain of tools to use (so you can inspect what it's doing), and another mechanism would be responsible for actually applying them to the input to y…

In your chain of tools are any of the tools themselves LLMs? Because that's the same problem except now you need to hijack the "parent" LLM to forward some malicious instructions down. And even if not, as long as there's any _execution_ or _write_ happening, the input could still modify the chain of tools being used. So you'd need _heavy_ restrictions on what the chains can actually do. How that intersects with opera…

Well, in the one-LLM case, the input would have no effect on the chain: you'd presumably describe the input format to the LLM, maybe with a few hand-picked example lines, and it would come up with a chain that should be untainted. In the two-LLM case, the chain generated by the ephemeral LLM would have to be considered tainted until proven otherwise. Your "LLM-in-the-loop" case would just be invariably asking for trouble.

Of course, the generated chain being buggy and vulnerable would also be an issue, since it would be less likely to be built with a posture of heavy validation. And in any case, the average user would rather just run on vibes rather than taking all these paranoid precautions. Then again, what do I know, maybe free-wheeling agents really will be everything they're hyped up to be in spite of the problems.

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#78

Post by simonw »

soulofmischief wrote: post #73
simonw wrote: post #48

Earlier quoted context omitted.

Have you seen that implemented yet?

Oh hey Simon! I independently landed on the same architecture in a prior startup before you published your dual LLM blog post, though unfortunately there's nothing left standing to show since that company experienced a hostile board takeover, the board squeezed me out of my CTO position in order to plant a yes man, pivoted to something I was against, and then recently shut down after failing to find product-market fi…

That's awesome to hear! I was never sure if anyone had managed to get it working.

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#79

Post by soulofmischief »

simonw wrote: post #78
soulofmischief wrote: post #73

Earlier quoted context omitted.

Oh hey Simon! I independently landed on the same architecture in a prior startup before you published your dual LLM blog post, though unfortunately there's nothing left standing to show since that company experienced a hostile board takeover, the board squeezed me out of my CTO position in order to plant a yes man, pivoted to something I was against, and then recently shut down after failing to find product-market fi…

That's awesome to hear! I was never sure if anyone had managed to get it working.

Not quite the same, but OpenAI is doing it in the opposite direction with their thinking models, hiding the reasoning step from the user and only providing a summarization. Maybe in the future, hosted agents have an airlock in both directions.

> ... in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. We also do not want to make an unaligned chain of thought directly visible to users.

> Therefore, after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users.

Source: https://openai.com/index/learning-to-reason-with-llms/

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#80

Post by dandelion9 »

breppp wrote: post #4

it uses all the jargon from real security (spraying, scope violation, bypass) but when reading these, it always sounds simple like essentially prompt injection, rather than some highly crafted shell code and unsafe memory exploitation

Your cited examples all make sense in the context of the article. How is a zero-click exfiltration of sensitive data vuln not "real security"?

Specialists require nuanced language when building up a body of research, in order to map out the topic and better communicate with one another.

Post reply on HN