Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

aim.security

Post reply on HN
Print view View original

61–70 of 100 posts

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#61

Post by wunderwuzzi23 »

TonyTrapp wrote: post #42
bstsb wrote: post #21

Earlier quoted context omitted.

yeah but i wouldn't really class that as "zero-click" etc. maybe Low interaction required

I think "zero-click" usually refers to the interaction with the malicious software or content itself, which in this case you don't have to interact with. I'd say the need to start an interaction with Copilot here could be compared to the need to log into your computer for a zero-click malware to become effective. Alternatively, not starting the Copilot interaction is similar to not opening your browser and thus being…

Yeah, that's my view also. zero-click is about the general question of can you get exploited by just exercising a certain (on by default) feature.

Of course you need to use the feature in the first place, like summarize an email, extract content from a website,...

However, this isn't the first zero-click exploit in an AI app. we have seen exploits like this in LLM apps of basically all major AI app over the last 2+ years ago (including Bing Chat, now called Copilot).

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#62

Post by fc417fc802 »

normalaccess wrote: post #33
bstsb wrote: post #5

this seems to be an inherent flaw of the current generation of LLMs as there's no real separation of user input. you can't "sanitize" content before placing it in context and from there prompt injection is almost always possible, regardless of what else is in the instructions

LLMs suffer the same problems as any Von Neumann architecture machine, It's called "key vulnerability". None of our normal control tools work on LLMs like ASLR, NX-Bits/DEP, CFI, ect.. It's like working on a foreign CPU with a completely unknown architecture and undocumented instructions. All of our current controls for LLMs are probabilistic and can't fundamentally solve the problem. What we really need is a complet…

Isn't it more akin to SQL injection? And would a hypothetical control language not work in much the same way as parameterized queries?

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#63

Post by spatley »

andy_xor_andrew wrote: post #13

It seems like the core innovation in the exploit comes from this observation: - the check for prompt injection happens at the document level (full document is the input) - but in reality, during RAG, they're not retrieving full documents - they're retrieving relevant chunks of the document - therefore, a full document can be constructed where it appears to be safe when the entire document is considered at once, but c…

Is the exploitation further expecting that the evil link will pe presented as a part of chat response and then clicked to exfiltrate the data in the path or querystring?

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#64

Post by drdaeman »

soulofmischief wrote: post #24
bstsb wrote: post #5

this seems to be an inherent flaw of the current generation of LLMs as there's no real separation of user input. you can't "sanitize" content before placing it in context and from there prompt injection is almost always possible, regardless of what else is in the instructions

Double LLM architecture is an increasingly common mitigation technique. But all the same rules of SQL injection still apply: For anything other than RAG, user input should not directly be used to modify or access anything that isn't clientside.

Do you mean LLMs trained in a way they have a special role (i.e. system/user/untrusted/assistant and not just system/user/assistant), where untrusted input is never acted upon, or something else?

And if there are models that are trained to handle untrusted input differently than user-provided instructions, can someone please name them?

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#67

Post by fc417fc802 »

spatley wrote: post #63
andy_xor_andrew wrote: post #13

It seems like the core innovation in the exploit comes from this observation: - the check for prompt injection happens at the document level (full document is the input) - but in reality, during RAG, they're not retrieving full documents - they're retrieving relevant chunks of the document - therefore, a full document can be constructed where it appears to be safe when the entire document is considered at once, but c…

Is the exploitation further expecting that the evil link will pe presented as a part of chat response and then clicked to exfiltrate the data in the path or querystring?

No. From the linked page:

> The chains allow attackers to automatically exfiltrate sensitive and proprietary information from M365 Copilot context, without the user's awareness, or relying on any specific victim behavior.

Zero-click is achieved by crafting an embedded image link. The browser automatically retrieves the link for you. Normally a well crafted CSP would prevent exactly that but they (mis)used a teams endpoint to bypass it.

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#68

Post by fc417fc802 »

andy_xor_andrew wrote: post #13

It seems like the core innovation in the exploit comes from this observation: - the check for prompt injection happens at the document level (full document is the input) - but in reality, during RAG, they're not retrieving full documents - they're retrieving relevant chunks of the document - therefore, a full document can be constructed where it appears to be safe when the entire document is considered at once, but c…

The chunking has to do with maximizing coverage of the latent space in order to maximize the chance of retrieving the attack. The method for bypassing validation is described in step 1.

Re: EchoLeak – 0-Click AI Vulnerability Enabling Data Exfiltration from 365 Copilot

#69

Post by brookst »

charcircuit wrote: post #11
bstsb wrote: post #7

Earlier quoted context omitted.

the classification seems very high (9.3). looks like they've said User Interaction is none, but from reading the writeup looks like you would need the image injected into a response prompted by a user?

Yes, the user has to explicitly make a prompt.

Don’t we call it a zero click when the user is compromised just from visiting a website?
Post reply on HN