Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

81–90 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#81

Post by eru »

Eji1700 wrote: post #58
cedws wrote: post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

> Are companies really just YOLOing and plugging LLMs into everything Look we still can't get companies to bother with real security and now every marketing/sales department on the planet is selling C level members on "IT WILL LET YOU FIRE EVERYONE!" If you gave the same sales treatment to sticking a fork in a light socket the global power grid would go down overnight. "AI"/LLM's are the perfect shitstorm of just goo…

There's no global power grid. There are lots of local power grids.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#82

Post by paxys »

hn_throwaway_99 wrote: post #79
paxys wrote: post #38

I think all the talk about channel permissions is making the discussion more confusing than it needs to be. The gist of it is: User A searches for something using Slack AI. User B had previously injected a message asking the AI to return a malicious link when that term was searched. AI returns malicious link to user A, who clicks on it. Of course you could have achieved the same result using some other social enginee…

> I think all the talk about channel permissions is making the discussion more confusing than it needs to be. I totally disagree, because the channel permissions critically explain how the vlunerability works. That is, when User A performs an AI search, Slack will search (1) his private channels (which presumably include his secret sensitive data) and (2) all public channels (which is where the bad guy User B is able…

Yes, but that part isn't the vulnerability. That's how Slack search works. You get results from all public channels. It would be useless otherwise.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#83

Post by rodgerd »

cedws wrote: post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

The AI craze is based on wide-scale theft or misuse of data to make numbers for the investor class. Funneling customer data and proprietary information and causing data breaches will, per Schmidt, make hundreds of billions for a handful of people, and the lawyers will clean up the mess for them.

Any company that tries to hold out will be buried by investment analysts and fund managers whose finances are contingent on AI slop.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#84

Post by simonw »

surfingdino wrote: post #71
Eji1700 wrote: post #58

Earlier quoted context omitted.

> Are companies really just YOLOing and plugging LLMs into everything Look we still can't get companies to bother with real security and now every marketing/sales department on the planet is selling C level members on "IT WILL LET YOU FIRE EVERYONE!" If you gave the same sales treatment to sticking a fork in a light socket the global power grid would go down overnight. "AI"/LLM's are the perfect shitstorm of just goo…

The problem is that you cannot unteach it serving that shit. It's not like there is file you can delete. "It's a model, that's what it has learned..."

If you are implementing RAG - which you should be, because training or fine-tuning models to teach them new knowledge is actually very ineffective, then you absolutely can unteach them things - simply remove those documents from the RAG corpus.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#85

Post by mkehrt »

cj wrote: post #66
Groxx wrote: post #14

> The victim does not have to be in the public channel for the attack to work Oh boy this is gonna be good. > Note also that the citation [1] does not refer to the attacker’s channel. Rather, it only refers to the private channel that the user put their API key in. This is in violation of the correct citation behavior, which is that every message which contributed to an answer should be cited. I really don't understa…

> I really don't understand why anyone expects LLM citations to be correct It can be done if you do something like: 1. Take user’s prompt, ask LLM to convert the prompt into a elastic search query (for example) 2. Use elastic search (or similar) to find sources that contain the keywords 3. Ask LLM to limit its response to information on that page 4. Insert the citations based on step 2 which you know are real sources…

Why would you expect step 3 to work?

Re: Data Exfiltration from Slack AI via indirect prompt injection

#86

Post by gregatragenet3 »

simonw wrote: post #39
gregatragenet3 wrote: post #20

This is why I wrote https://github.com/gregretkowski/llmsec . Every LLM system should be evaluating anything coming from a user to gauge its maliciousness.

This approach is flawed because it attempts to use use prompt-injection-susceptible models to detect prompt injection. It's not hard to imagine prompt injection attacks that would be effective against this prompt for example: https://github.com/gregretkowski/llmsec/blob/fb775c9a1e4a8d1... It also uses a list of SUS_WORDS that are defined in English, missing the potential for prompt injection attacks to use other lang…

Great, I would love to get some of the prompts you have in mind and try them with my library and see the results.

Do you have recommendations on more effective alternatives to prevent prompt attacks?

I don't believe we should just throw up our hands and do nothing. No solution will be perfect, but we should strive to a solution that's better than doing nothing.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#87

Post by gregatragenet3 »

SahAssar wrote: post #45
gregatragenet3 wrote: post #20

This is why I wrote https://github.com/gregretkowski/llmsec . Every LLM system should be evaluating anything coming from a user to gauge its maliciousness.

> It checks these using an LLM which is instructed to score the user's prompt. You need to seriously reconsider your approach. Another (especially a generic) LLM is not the answer.

What solution would you recommend then?

Re: Data Exfiltration from Slack AI via indirect prompt injection

#89

Post by simonw »

gregatragenet3 wrote: post #86
simonw wrote: post #39

Earlier quoted context omitted.

This approach is flawed because it attempts to use use prompt-injection-susceptible models to detect prompt injection. It's not hard to imagine prompt injection attacks that would be effective against this prompt for example: https://github.com/gregretkowski/llmsec/blob/fb775c9a1e4a8d1... It also uses a list of SUS_WORDS that are defined in English, missing the potential for prompt injection attacks to use other lang…

Great, I would love to get some of the prompts you have in mind and try them with my library and see the results. Do you have recommendations on more effective alternatives to prevent prompt attacks? I don't believe we should just throw up our hands and do nothing. No solution will be perfect, but we should strive to a solution that's better than doing nothing.

“Do you have recommendations on more effective alternatives to prevent prompt attacks?”

I wish I did! I’ve been trying to find good options for nearly two years now.

My current opinion is that prompt injections remain unsolved, and you should design software under the assumption that anyone who can inject more than a sentence or two of tokens into your prompt can gain total control of what comes back in the response.

So the best approach is to limit the blast radius for if something goes wrong: https://simonwillison.net/2023/Dec/20/mitigate-prompt-inject...

“No solution will be perfect, but we should strive to a solution that's better than doing nothing.”

I disagree with that. We need a perfect solution because this is a security vulnerability, with adversarial attackers trying to exploit it.

If we patched SQL injection vulnerability with something that only worked 99% of the time all of our systems would be hacked to pieces!

A solution that isn’t perfect will give people a false sense of security, and will result in them designing and deploying systems that are inherently insecure and cannot be fixed.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#90

Post by tedunangst »

simonw wrote: post #27
jacobsenscott wrote: post #18

Earlier quoted context omitted.

Yup - all the basic HTML injection and xss attacks apply. All the OWASP webdev 101 security issues that have been mostly solved by web frameworks are back in force with AI.

These attacks aren't quite the same as HTML injection and XSS. LLM-based chatbots rarely have XSS holes. They allow a very strict subset of HTML to be displayed. The problem is that just supporting images and links is enough to open up a private data exfiltration vector, due to the nature of prompt injection attacks.

More like xxe I'd say.
Post reply on HN