Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

101–110 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#101
post #98
post #84

Earlier quoted context omitted.

If you are implementing RAG - which you should be, because training or fine-tuning models to teach them new knowledge is actually very ineffective, then you absolutely can unteach them things - simply remove those documents from the RAG corpus.

I still don't understand the hype behind rag. Like yeah it's a natural language interface into whatever database is being integrated, but is that actually worth the billions being spent here? I've heard they still hallucinate even when you are using rag techniques.

Being able to ask a question in human language and get back an answer is the single most useful thing that LLMs have to offer.

The obvious challenge here is "how do I ensure it can answer questions about this information that wasn't included in its training data?"

RAG is the best answer we have to that. Done well it can work great.

(Actually doing it well is surprisingly difficult - getting a basic implementation of RAG up and running is a couple of hours of hacking, making it production ready against whatever weird things people might throw at it can take months.)

Re: Data Exfiltration from Slack AI via indirect prompt injection

#102
post #21

Earlier quoted context omitted.

Yup - all the basic HTML injection and xss attacks apply. All the OWASP webdev 101 security issues that have been mostly solved by web frameworks are back in force with AI.

Can’t upvote you enough on this point. It’s like everyone lost their collective mind and forgot the lessons of the past twenty years.

This presents an incredible opportunity. The problems are known. The solutions somewhat. Now make a business selling the solution.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#103
post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

Yeah, the thing that took me a bit to understand is that, when you do a search (or AI does a search for you) in Slack, it will search: 1. All public channels 2. Any private channels that only you have access to. That permissions model is still intact, and that's not what is broken here. What's going on is a malicious actor is using a public channel to essentially do prompt injection, so then when another user does a…

As a developer I learned a long time ago that if I didn't understand how something worked, I shouldn't use it in production code. I can barely follow this scenario, I don't understand how AI does what it does (I think even the people who invented it don't really understand how it works) so it's something I would never bake into anything I create.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#104
post #44
post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

The S in LLM stands for safety!

Or Security.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#106

Wouldn't it be better to put "confetti" -- the API key as part of the domain name? That way, the key would be leaked without any required clicks due to the DNS prefetching by the browser.

How would you own the server if you don't know what the domain is going to be? Perhaps I don't understand. Edit: Ah, wildcard subdomain? Does that get prefetched in Slack? Pretty terrible if so.

Subdomains.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#107
post #81
post #58

Earlier quoted context omitted.

> Are companies really just YOLOing and plugging LLMs into everything Look we still can't get companies to bother with real security and now every marketing/sales department on the planet is selling C level members on "IT WILL LET YOU FIRE EVERYONE!" If you gave the same sales treatment to sticking a fork in a light socket the global power grid would go down overnight. "AI"/LLM's are the perfect shitstorm of just goo…

There's no global power grid. There are lots of local power grids.

Pedantically, yes, but it doesn't really matter to OP's real message: The problematic effect would be global in scope, as people everywhere would do stupid things to an arbitrary number of discrete grids or generation systems.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#109
post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

For bots in Slack, Discord, Teams, Telegram,... there is actually another exfiltration vector called "unfurling"!

All an attacker has to do is render a hyperlink, no clicking needed. I discussed this and how to mitigate it here: https://embracethered.com/blog/posts/2024/the-dangers-of-unf...

So, hopefully Slack AI does not automatically unfurl links...

Re: Data Exfiltration from Slack AI via indirect prompt injection

#110

Slack’s response here is alarming. If I’m getting the PoC correctly, this is data exfil from private channels, not public ones as their response seems to suggest. I’d want to know if you can prompt the AI to exfil data from private channels where the prompt author isn’t a member.

What's happening here is you can make the slack AI hallucinate a message that never existed by telling it to combine your private messages with another message in a public channel in arbitrary ways. Slack claims it isn't a problem because the user doing the "ai assisted" search has permission to both the private and public data. However that data never existed in the format the AI responds with . An attacker can make…

Quick fix is pull the AI. Or minimum rip out any links it provides. If it needs to link it can refer to the slack message that has the necessary info, which could still be harmful (non AI problem there) but cannot exfil like this.
Post reply on HN