Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

71–80 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#71

Post by surfingdino »

Eji1700 wrote: post #58
cedws wrote: post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

> Are companies really just YOLOing and plugging LLMs into everything Look we still can't get companies to bother with real security and now every marketing/sales department on the planet is selling C level members on "IT WILL LET YOU FIRE EVERYONE!" If you gave the same sales treatment to sticking a fork in a light socket the global power grid would go down overnight. "AI"/LLM's are the perfect shitstorm of just goo…

The problem is that you cannot unteach it serving that shit. It's not like there is file you can delete. "It's a model, that's what it has learned..."

Re: Data Exfiltration from Slack AI via indirect prompt injection

#73

Post by Groxx »

cj wrote: post #66
Groxx wrote: post #14

> The victim does not have to be in the public channel for the attack to work Oh boy this is gonna be good. > Note also that the citation [1] does not refer to the attacker’s channel. Rather, it only refers to the private channel that the user put their API key in. This is in violation of the correct citation behavior, which is that every message which contributed to an answer should be cited. I really don't understa…

> I really don't understand why anyone expects LLM citations to be correct It can be done if you do something like: 1. Take user’s prompt, ask LLM to convert the prompt into a elastic search query (for example) 2. Use elastic search (or similar) to find sources that contain the keywords 3. Ask LLM to limit its response to information on that page 4. Insert the citations based on step 2 which you know are real sources…

It also doesn't prevent it from hallucinating something wholesale from the rest of the corpus it was trained on. Sometimes this is a huge source of incorrect results due to almost-but-not-quite matching public data.

But yes, a complete list of "we fed it this" is useful and relatively trustworthy in ways that "ask the LLM to cite what it used" is absolutely not.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#74

Post by Groxx »

paxys wrote: post #38

I think all the talk about channel permissions is making the discussion more confusing than it needs to be. The gist of it is: User A searches for something using Slack AI. User B had previously injected a message asking the AI to return a malicious link when that term was searched. AI returns malicious link to user A, who clicks on it. Of course you could have achieved the same result using some other social enginee…

There's an important step missing in this summary: Slack AI adds the user's private data to the malicious link, because the injected link doesn't contain that.

That it also cites it as "this came from your slack messages" is just a cherry on top.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#76

Post by hn_throwaway_99 »

simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

Yeah, the thing that took me a bit to understand is that, when you do a search (or AI does a search for you) in Slack, it will search:

1. All public channels

2. Any private channels that only you have access to.

That permissions model is still intact, and that's not what is broken here. What's going on is a malicious actor is using a public channel to essentially do prompt injection, so then when another user does a search, the malicious user still doesn't have access to any of that data, but the prompt injection tricks the AI result for the original "good" user to be a link to the malicious user's website - it basically is an AI-created phishing attempt at that point.

Looking through the details I think it would be pretty difficult to actually exploit this vulnerability in the real world (because the malicious prompt injection, created beforehand, would need to match fairly closely what the good user would be searching for), but just highlights the "Alice in Wonderland" world of LLM prompt injections, where it's essentially impossible to separate instructions from data.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#78

Post by aftbit »

Jenk wrote: post #69
samstave wrote: post #48

Earlier quoted context omitted.

Some ( all ) CEOs learned that offering a free month coupon/voucher for Future Security Services to secure your information against a breach like the one that just happened on the platform that's offering you a free voucher to secure your data that sits on the platform that was compromised and leaked your data, is a nifty-clean way to handle such legal inconveniences. Oh, and some supposed financial penalty is claime…

Techies aren't immune either, before we all follow the "blame management" bandwagon for the 2^101-tieth time. CEOs aren't the reason supply chain attacks are absolutely rife with problems right now. That's entirely on the technical experts who created all of those pinnacle achievements in tech ranging from tech-led orgs and open source community built package ecosystems. Arbitrary code execution in homebrew, scoop, c…

Yeah supply chain stuff is scary and still very open. This ranges from the easy stuff like typo-squatting pip packages or hacktavists changing their npm packages to wreck all computers in Russia up to the advanced backdoors like the xz hack.

Another big still mostly open category is speculative execution data leaks or other "abstraction breaks" like Rowhammer.

At least in theory things like Passkeys and ubiquitous password manager use should eventually start to cut down on simple phishing attacks.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#79

Post by hn_throwaway_99 »

paxys wrote: post #38

I think all the talk about channel permissions is making the discussion more confusing than it needs to be. The gist of it is: User A searches for something using Slack AI. User B had previously injected a message asking the AI to return a malicious link when that term was searched. AI returns malicious link to user A, who clicks on it. Of course you could have achieved the same result using some other social enginee…

> I think all the talk about channel permissions is making the discussion more confusing than it needs to be.

I totally disagree, because the channel permissions critically explain how the vlunerability works. That is, when User A performs an AI search, Slack will search (1) his private channels (which presumably include his secret sensitive data) and (2) all public channels (which is where the bad guy User B is able to put a message that does the prompt injection), importantly including ones that User A has never joined and has never seen.

That is, the only reason this vulnerability works is because User B is able to create a public channel but with himself as the only user so that it's highly unlikely anyone else would find it.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#80

Post by ryoshu »

cedws wrote: post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

Yes. And no one wants to listen to the people who deal with this for a living.
Post reply on HN