Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

171–180 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#171

Post by gregatragenet3 »

__loam wrote: post #127
simonw wrote: post #101

Earlier quoted context omitted.

Being able to ask a question in human language and get back an answer is the single most useful thing that LLMs have to offer. The obvious challenge here is "how do I ensure it can answer questions about this information that wasn't included in its training data?" RAG is the best answer we have to that. Done well it can work great. (Actually doing it well is surprisingly difficult - getting a basic implementation of…

I recognize it's useful. I don't think it justifies the cost.

What cost? A few cents per question answered?

Re: Data Exfiltration from Slack AI via indirect prompt injection

#172

Post by jesprenj »

reassess_blind wrote: post #96
jerjerjer wrote: post #94

Earlier quoted context omitted.

Wildcard dns would work: *.example.com. 14400 IN A 1.2.3.4 after that just collect webserver logs.

Yeah, assuming Slack does prefetch these links that makes the attack significantly easier and faster to carry out.

I actually meant DNS prefetching, not HTTP prefetching. I don't think browsers will prefetch (make HTTP GET requests before they are clicked) links by default (maybe slack does to get metadata), but they quite often prefetch the DNS host records as soon as an "a href" appears.

In case of DNS prefetching, a wildcard record wouldn't be needed, you just need to control the nameservers of the domain and enable query logging.

But I'm not sure how do browsers decide what links to DNS prefetch, maybe it's not even possible for links generated with JS or something like that ... I'm just guessing.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#173

Post by simonw »

surfingdino wrote: post #169
neverokay wrote: post #155

Earlier quoted context omitted.

Maybe. I think users will be largely in control of their context and message history over the course of decades. Context is not being stored in Gemini or OpenAi (yet, I think, not to that degree). My one year’s worth of LLM chats isn’t actually stored anywhere yet and doesn’t have to be, and for the most part I’d want it to be portable. I’d say this is probably something that needs to be legally protected asap.

My trust in AI operators not storing original content for later use is zero.

If you pay them enough money you can sign a custom contract with them that means you can sue them to pieces if they are later found to be storing your original content despite saying that they aren't.

Personally I've decided to trust them when they tell me they won't do that in their terms and conditions. My content isn't actually very valuable to them.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#174

Post by simonw »

troyvit wrote: post #166

I suck at security, let's get this out of the way. However, it seems like to make this exfiltration work you need access to the Slack workspace. In other words the malicious user is already operating from within. I see two possibilities of how that would happen. Either you're already a member of the organization and you want to burn it all down, or you broke the security model of an organization and you are in their…

I've seen plenty of organizations who run community Slack channels where they invite non-employees in to talk with them - I'm a member of several of those myself.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#175

Post by troyvit »

simonw wrote: post #174
troyvit wrote: post #166

I suck at security, let's get this out of the way. However, it seems like to make this exfiltration work you need access to the Slack workspace. In other words the malicious user is already operating from within. I see two possibilities of how that would happen. Either you're already a member of the organization and you want to burn it all down, or you broke the security model of an organization and you are in their…

I've seen plenty of organizations who run community Slack channels where they invite non-employees in to talk with them - I'm a member of several of those myself.

Hm that's a good point, and we've done that ourselves. I believe we limited those folks to one private channel and didn't allow them to create new channels.

I think of it like an office space. If you bring in some consultants do you set up a space for them and keep them off your VPN, or do you let them run around, sit where they want, and peek over everybody's shoulder to see what they're up to?

Re: Data Exfiltration from Slack AI via indirect prompt injection

#176

Post by titzer »

cedws wrote: post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

The whole idea that we're going to build software systems using natural language prompts to AI models which then promptly (heh) fall on their face because they mash together text strings to feed to a huge inscrutable AI is lazy and stupid. We're in a dumb future where "SUDO make me a sandwich" is a real attack strategy.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#177

Post by michaelmior »

fsndz wrote: post #147
michaelmior wrote: post #145

Earlier quoted context omitted.

They have to be part of the same Slack workspace, but not necessarily the same organization.

yeah so the same company. and given the type of attack have to have a lot of knowledge about usernames and what they may have potentially shared in some random private slack channel. I can understand why slack is not alarmed with this. would like to see their official response though

Same workspace != same company. It's not uncommon to have people from multiple organizations in the same workspace.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#178

Post by simonw »

troyvit wrote: post #175
simonw wrote: post #174

Earlier quoted context omitted.

I've seen plenty of organizations who run community Slack channels where they invite non-employees in to talk with them - I'm a member of several of those myself.

Hm that's a good point, and we've done that ourselves. I believe we limited those folks to one private channel and didn't allow them to create new channels. I think of it like an office space. If you bring in some consultants do you set up a space for them and keep them off your VPN, or do you let them run around, sit where they want, and peek over everybody's shoulder to see what they're up to?

The bigger problem here is that Slack AI has a misfeature where malicious instructions can cause it to answer questions with links that leak data. The specific examples aren't as important as the overall class of attack.

Anything you say in Slack - or anything in a document that is available within Slack - could potentially be leaked to an attacker who manages to get their malicious instructions into your Slack. There are many ways they might be able to do that, such as tricking an employee of yours into uploading a file to Slack that includes those instructions.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#179

Post by fsndz »

simonw wrote: post #165
fsndz wrote: post #163

Earlier quoted context omitted.

a belief is not the truth

So they shouldn’t have published what they’ve discovered so far?

I think it was great that they contacted Slack the way they did. It's also okay for me to publish. I just don't think it deserves much fanfare; in my opinion, this isn't a huge or serious vulnerability, that's all.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#180

Post by fsndz »

michaelmior wrote: post #177
fsndz wrote: post #147

Earlier quoted context omitted.

yeah so the same company. and given the type of attack have to have a lot of knowledge about usernames and what they may have potentially shared in some random private slack channel. I can understand why slack is not alarmed with this. would like to see their official response though

Same workspace != same company. It's not uncommon to have people from multiple organizations in the same workspace.

This makes the described attack seem even less interesting/dangerous. Thanks
Post reply on HN