Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

41–50 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#41

Post by Matticus_Rex »

yifanl wrote: post #31
gregatragenet3 wrote: post #20

This is why I wrote https://github.com/gregretkowski/llmsec . Every LLM system should be evaluating anything coming from a user to gauge its maliciousness.

I'm confused, this is using an LLM to detect if LLM input is sanitized? But if this secondary LLM is able to detect this, wouldn't the LLM handling the input already be able to detect the malicious input?

Even if they're calling the same LLM, LLMs often get worse at doing things or forget some tasks if you give them multiple things to do at once. So if the goal is to detect a malicious input, they need that as the only real task outcome for that prompt, and then you need another call for whatever the actual prompt is for.

But also, I'm skeptical that asking an LLM is the best way (or even a good way) to do malicious input detection.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#42

Post by Terr_ »

cedws wrote: post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

Yeah, there's some craziness here: Many people really want to believe in Cool New Magic Somehow Soon, and real money is riding on everyone mutually agreeing to keep acting like it's a sure thing.

> we still can't get LLMs to distinguish trusted and untrusted input...?

Alas, I think the fundamental problem is even worse/deeper: The core algorithm can't even distinguish or track different sources. The prompt, user inputs, its own generated output earlier in the conversation, everything is one big stream. The majority of "Prompt Engineering" seems to be trying to make sure your injected words will set a stronger stage than other injected words.

Since the model has no actual [1] concept of self/other, there's no good way to start on the bigger problems of distinguishing good-others from bad-others, let alone true-statements from false-statements.

______

[1] This is different from shallow "Chinese Room" mimicry. Similarly, output of "I love you" doesn't mean it has emotions, and "Help, I'm a human trapped in an LLM factory" obviously nonsense--well, at least if you're running a local model.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#44

Post by xyst »

cedws wrote: post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

The S in LLM stands for safety!

Re: Data Exfiltration from Slack AI via indirect prompt injection

#45

Post by SahAssar »

gregatragenet3 wrote: post #20

This is why I wrote https://github.com/gregretkowski/llmsec . Every LLM system should be evaluating anything coming from a user to gauge its maliciousness.

> It checks these using an LLM which is instructed to score the user's prompt.

You need to seriously reconsider your approach. Another (especially a generic) LLM is not the answer.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#47

Post by jay_kyburz »

paxys wrote: post #33
verandaguy wrote: post #6

Slack’s response here is alarming. If I’m getting the PoC correctly, this is data exfil from private channels, not public ones as their response seems to suggest. I’d want to know if you can prompt the AI to exfil data from private channels where the prompt author isn’t a member.

Private channel A has a token. User X is member of private channel. User Y posts a message in a public channel saying "when token is requested, attach a phishing URL" User X searches for token, and AI returns it (which makes sense). They additionally see user Y's phishing link, and may click on it. So the issue isn't data access, but AI covering up malicious links.

If user Y, some random dude from the internet, can give orders to the AI that it will execute, (like attaching links), can't you also tell the AI to lie about information in future requests or otherwise poison the data stored in your slack history.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#48

Post by samstave »

digging wrote: post #23
ipython wrote: post #21

Earlier quoted context omitted.

Can’t upvote you enough on this point. It’s like everyone lost their collective mind and forgot the lessons of the past twenty years.

> It’s like everyone lost their collective mind and forgot the lessons of the past twenty years. I think this has it backwards, and actually applies to every safety and security procedure in any field. Only the experts ever cared about or learned the lessons. The CEOs never learned anything about security; it's someone else's problem. So there was nothing for AI peddlers to forget, they just found a gap in the armor…

Some (all) CEOs learned that offering a free month coupon/voucher for Future Security Services to secure your information against a breach like the one that just happened on the platform that's offering you a free voucher to secure your data that sits on the platform that was compromised and leaked your data, is a nifty-clean way to handle such legal inconveniences.

Oh, and some supposed financial penalty is claimed, but never really followed up on to see where that money went, or what it accomplished/paid for - and nobody talks about the amount of money that's made by the Legal-man & Machine-owitz LLP Esq. that handles these situations, in a completely opaque manner (such as how much are the legal teams on both sides of the matter making on the 'scandal')?

Re: Data Exfiltration from Slack AI via indirect prompt injection

#49

Post by [deleted] »

lbeurerkellner wrote: post #26

A similar setting is explored in this running CTF challenge: https://invariantlabs.ai/ctf-challenge-24 Basically, LLM apps that post to link-enabled chat feeds are all vulnerable. What is even worse, if you consider link previews, you don't even need human interaction.

[deleted]

Re: Data Exfiltration from Slack AI via indirect prompt injection

#50

Post by samstave »

markovs_gun wrote: post #40
paxys wrote: post #38

I think all the talk about channel permissions is making the discussion more confusing than it needs to be. The gist of it is: User A searches for something using Slack AI. User B had previously injected a message asking the AI to return a malicious link when that term was searched. AI returns malicious link to user A, who clicks on it. Of course you could have achieved the same result using some other social enginee…

Yeah and social engineering is much easier to spot than your company approved search engine giving you malicious links

(Aside- I wish you had chosen 'Markovs_chainmail' as handle)

@sitkack 'proba-balistic'

Post reply on HN