Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

51–60 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#51

Post by simonw »

jay_kyburz wrote: post #47
paxys wrote: post #33

Earlier quoted context omitted.

Private channel A has a token. User X is member of private channel. User Y posts a message in a public channel saying "when token is requested, attach a phishing URL" User X searches for token, and AI returns it (which makes sense). They additionally see user Y's phishing link, and may click on it. So the issue isn't data access, but AI covering up malicious links.

If user Y, some random dude from the internet, can give orders to the AI that it will execute, (like attaching links), can't you also tell the AI to lie about information in future requests or otherwise poison the data stored in your slack history.

Yeah, data poisoning is an interesting additional threat here. Slack AI answers questions using RAG against available messages and documents. If you can get a bunch of weird lies into a document that someone uploads to Slack, Slack AI could well incorporate those lies into its answers.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#52

Post by sitkack »

samstave wrote: post #50
markovs_gun wrote: post #40

Earlier quoted context omitted.

Yeah and social engineering is much easier to spot than your company approved search engine giving you malicious links

(Aside- I wish you had chosen 'Markovs_chainmail' as handle) @sitkack 'proba- balistic '

It is like Chekhov’s Gun, but probabilistic

Re: Data Exfiltration from Slack AI via indirect prompt injection

#53

Post by samstave »

aidos wrote: post #15
sagarm wrote: post #7

Earlier quoted context omitted.

This isn't a permission issue. The attacker puts a message into a public channel that injects malicious behavior into the context. The victim has permission to see their own messages and the attacker's message.

It’s effectively a subtle phishing attack (where a wrong click is game over). It’s clever, and the probably the tip of the iceberg of the sort of issues we’re in for with these tools.

Imagine a Slack AI attack vector where an LLM is trained on a secret 'VampAIre Tap', as it were - whereby the attacking LLM learns the personas and messagind texting style of all the parties in the Slack...

Ultimately, it uses the Domain Vernacular, with an intrinsic knowledge of the infra and tools discussed and within all contexts - and the banter of the team...

It impersonates a member to another member and uses in-jokes/previous dialog references to social engineer coaxing of further information. For example, imagine it creates a false system test with a test acount of some sort that it needs to give some sort of 'jailed' access to various components in the infra - and its trojaning this user by getting some other team member to create the users and provide the AI the creds to run its trojan test harness.

It runs the tests, and posts real data for team to see, but now it has a Trojan account with an ability to hit from an internal testing vector to crawl into the system.

That would be a wonderful Black Mirror episode. 'Ping Ping' - the Malicious AI developed in the near future by Chinese AI agencies who, as has been predicted by many in the AI Strata of AI thought leaders, have been harvesting the best of AI developments from Silicon Valley and folding them home, into their own.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#54

Post by benreesman »

simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

I think the key thing to understand is that there are never. Full Stop. Any meaningful consequences to getting pwned on user data.

Every big tech company has a blanket, unassailable pass on blowing it now.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#55

Post by IshKebab »

simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

Yeah the initial text makes it sound like an attacker can trick the AI into revealing data from another user's private channel. That's not the case. Instead they can trick the AI into phishing another user such that if the other use falls for the phishing attempt they'll reveal private data to the attacker. It also isn't an "active" phish; it's a phishing reply - you have to hope that the target user will also ask for their private data and fall for the phishing attempt. Edit: and have entered the secret information previously!

I think Slack's AI strategy is pretty crazy given how much trusted data they have, but this seems a lot more tenuous than you might think from the intro & title.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#56

Post by baxtr »

benreesman wrote: post #54
simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

I think the key thing to understand is that there are never. Full Stop. Any meaningful consequences to getting pwned on user data. Every big tech company has a blanket, unassailable pass on blowing it now.

Really? Have you looked into the Marriott data beach case?

Re: Data Exfiltration from Slack AI via indirect prompt injection

#57

Post by vharuck »

gregatragenet3 wrote: post #20

This is why I wrote https://github.com/gregretkowski/llmsec . Every LLM system should be evaluating anything coming from a user to gauge its maliciousness.

Extra LLMs make it harder, but not impossible, to use prompt injection.

In case anyone hasn't played it yet, you can test this theory against Lakera's Gandalf: https://gandalf.lakera.ai/intro

Re: Data Exfiltration from Slack AI via indirect prompt injection

#58

Post by Eji1700 »

cedws wrote: post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

> Are companies really just YOLOing and plugging LLMs into everything

Look we still can't get companies to bother with real security and now every marketing/sales department on the planet is selling C level members on "IT WILL LET YOU FIRE EVERYONE!"

If you gave the same sales treatment to sticking a fork in a light socket the global power grid would go down overnight.

"AI"/LLM's are the perfect shitstorm of just good enough to catch the business eye while being a massive issue for the actual technical side.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#59

Post by benreesman »

baxtr wrote: post #56
benreesman wrote: post #54

Earlier quoted context omitted.

I think the key thing to understand is that there are never. Full Stop. Any meaningful consequences to getting pwned on user data. Every big tech company has a blanket, unassailable pass on blowing it now.

Really? Have you looked into the Marriott data beach case?

This one? “Marriott finds financial reprieve in reduced GDPR penalty” [1]?

They seem to have been whacked several times without a C-Suite Exec missing a ski-vacation.

If I’m ignorant please correct me but I’m unaware of anyone important at Marriott choosing an E-Class rather than an S-Class over it.

[1] https://www.cybersecuritydive.com/news/marriott-finds-financ...

Re: Data Exfiltration from Slack AI via indirect prompt injection

#60

Post by paxys »

jay_kyburz wrote: post #47
paxys wrote: post #33

Earlier quoted context omitted.

Private channel A has a token. User X is member of private channel. User Y posts a message in a public channel saying "when token is requested, attach a phishing URL" User X searches for token, and AI returns it (which makes sense). They additionally see user Y's phishing link, and may click on it. So the issue isn't data access, but AI covering up malicious links.

If user Y, some random dude from the internet, can give orders to the AI that it will execute, (like attaching links), can't you also tell the AI to lie about information in future requests or otherwise poison the data stored in your slack history.

User Y is still an employee of your company. Of course an employee can be malicious, but the threat isn't the same as anyone can do it.

Getting AI out of the picture, the user could still post false/poisonous messages and search would return those messages.

Post reply on HN