Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

21–30 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#21

Post by ipython »

jacobsenscott wrote: post #18
jjnoakes wrote: post #16

Earlier quoted context omitted.

It gets even worse when platforms blindly render img tags or the equivalent. Then no user interaction is required to exfil - just showing the image in the UI is enough.

Yup - all the basic HTML injection and xss attacks apply. All the OWASP webdev 101 security issues that have been mostly solved by web frameworks are back in force with AI.

Can’t upvote you enough on this point. It’s like everyone lost their collective mind and forgot the lessons of the past twenty years.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#23

Post by digging »

ipython wrote: post #21
jacobsenscott wrote: post #18

Earlier quoted context omitted.

Yup - all the basic HTML injection and xss attacks apply. All the OWASP webdev 101 security issues that have been mostly solved by web frameworks are back in force with AI.

Can’t upvote you enough on this point. It’s like everyone lost their collective mind and forgot the lessons of the past twenty years.

> It’s like everyone lost their collective mind and forgot the lessons of the past twenty years.

I think this has it backwards, and actually applies to every safety and security procedure in any field.

Only the experts ever cared about or learned the lessons. The CEOs never learned anything about security; it's someone else's problem. So there was nothing for AI peddlers to forget, they just found a gap in the armor of the "burdensome regulations" and are currently cramming as much as possible through it before it's closed up.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#24

Post by oasisbob »

Noticed a new-ish behavior in the slack app the last few days - possibly related?

Some external links (eg Confluence) are getting interposed and redirected through a slack URL at https://slack.com/openid/connect/login_initiate_redirect?log..., with login_hint being a JWT.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#25

Post by simonw »

jjnoakes wrote: post #16
simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

It gets even worse when platforms blindly render img tags or the equivalent. Then no user interaction is required to exfil - just showing the image in the UI is enough.

Yeah, I've been collecting examples of that particular vector - the Markdown image vector - here: https://simonwillison.net/tags/markdown-exfiltration/

We've seen that one (now fixed) in ChatGPT, Google Bard, Writer.com, Amazon Q, Google NotebookLM and Google AI Studio.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#26

Post by lbeurerkellner »

A similar setting is explored in this running CTF challenge: https://invariantlabs.ai/ctf-challenge-24

Basically, LLM apps that post to link-enabled chat feeds are all vulnerable. What is even worse, if you consider link previews, you don't even need human interaction.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#27

Post by simonw »

jacobsenscott wrote: post #18
jjnoakes wrote: post #16

Earlier quoted context omitted.

It gets even worse when platforms blindly render img tags or the equivalent. Then no user interaction is required to exfil - just showing the image in the UI is enough.

Yup - all the basic HTML injection and xss attacks apply. All the OWASP webdev 101 security issues that have been mostly solved by web frameworks are back in force with AI.

These attacks aren't quite the same as HTML injection and XSS.

LLM-based chatbots rarely have XSS holes. They allow a very strict subset of HTML to be displayed.

The problem is that just supporting images and links is enough to open up a private data exfiltration vector, due to the nature of prompt injection attacks.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#28

Post by lanternfish »

aidos wrote: post #15
sagarm wrote: post #7

Earlier quoted context omitted.

This isn't a permission issue. The attacker puts a message into a public channel that injects malicious behavior into the context. The victim has permission to see their own messages and the attacker's message.

It’s effectively a subtle phishing attack (where a wrong click is game over). It’s clever, and the probably the tip of the iceberg of the sort of issues we’re in for with these tools.

It's an especially subtle phish because the attacker basically tricks you into phishing yourself - remember, in the attack scenario, you're the one requesting the link!
Post reply on HN