Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

131–140 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#131

Post by Jenk »

sebastiennight wrote: post #122
Jenk wrote: post #69

Earlier quoted context omitted.

Techies aren't immune either, before we all follow the "blame management" bandwagon for the 2^101-tieth time. CEOs aren't the reason supply chain attacks are absolutely rife with problems right now. That's entirely on the technical experts who created all of those pinnacle achievements in tech ranging from tech-led orgs and open source community built package ecosystems. Arbitrary code execution in homebrew, scoop, c…

Wait, where can we read more about that? When you say "the keys to prod" do you mean the prod .ENV variables, or something else?

https://www.theverge.com/2023/2/28/23618353/lastpass-securit...

An employee (dev/sysadmin) had their home device compromised via a supply chain attack, which installed a keylogger and the attacker(s) were able to exfiltrate the credentials to lastpass cloud envs.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#132

Post by incorrecthorse »

Aren't you screwed from the moment you have a malicious user in your workspace? This user can change their picture/name and directly ask for the API key, or send some phishing link or get loose on whatever social engineering is fundamentally possible in any instant message system.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#133

Post by zelphirkalt »

wood_spirit wrote: post #112
SoftTalker wrote: post #103

Earlier quoted context omitted.

As a developer I learned a long time ago that if I didn't understand how something worked, I shouldn't use it in production code. I can barely follow this scenario, I don't understand how AI does what it does (I think even the people who invented it don't really understand how it works) so it's something I would never bake into anything I create.

Lots of coders use ai like copilot to develop code. This attack is like setting up lots of GitHub repos where the code is malicious and then the ai learning that that is how you routinely implement something basic and then generating that backdoored code when a trusting developer asks the ai how to implement login. Another parallel would be if yahoo gave their emails to ai. Their spam filtering is so bad that all the…

You can be responsibly using the current crop of ai to do coding, and you can do it recklessly: You can be diligently reading everything it writes for you and thinks about all the code and check, whether it just regurgitated some GPLed or AGPLed code, oooor ... you can be reckless and just use it. Moral choice of the user and immoral implementation of the creators of the ai.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#135

Post by namaria »

swyx wrote: post #64
jjmaxwell4 wrote: post #2

It's nuts how large and different the attack surfaces have gotten with AI

have they? as other comments mention this is the same attack surface as a regular phishing attack.

It's plainly not, when a phishing attack is receiving unsolicited links and providing compromising data, while this is getting it by asking the AI for something and getting a one-click attack injected in the answer.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#136

Post by namaria »

gregatragenet3 wrote: post #87
SahAssar wrote: post #45

Earlier quoted context omitted.

> It checks these using an LLM which is instructed to score the user's prompt. You need to seriously reconsider your approach. Another (especially a generic) LLM is not the answer.

What solution would you recommend then?

Don't graft generative AI on your system? Seems pretty straightforward to me.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#137

Post by h1fra »

incorrecthorse wrote: post #132

Aren't you screwed from the moment you have a malicious user in your workspace? This user can change their picture/name and directly ask for the API key, or send some phishing link or get loose on whatever social engineering is fundamentally possible in any instant message system.

There are a lot of public Slack for SaaS companies, phishing can be detected by serious users (especially when the messages seems phishy) but an indirect AI leak does not put you in a "defense mode", all it takes is one accidental click

Re: Data Exfiltration from Slack AI via indirect prompt injection

#138

Post by Eisenstein »

typeofhuman wrote: post #102
ipython wrote: post #21

Earlier quoted context omitted.

Can’t upvote you enough on this point. It’s like everyone lost their collective mind and forgot the lessons of the past twenty years.

This presents an incredible opportunity. The problems are known. The solutions somewhat. Now make a business selling the solution.

How do you 'undo' an entire market founded on fixing mistakes that shouldn't have been made once it gets established? Like the US tax system doesn't get some simple problems fixed because there are entire industries reliant upon them not getting fixed. I'm not sure encouraging outsiders to make a business model around patching over things that shouldn't be happening in the first place is the optimal way to solve the issues in the long term.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#139

Post by [deleted] »

wunderwuzzi23 wrote: post #109
simonw wrote: post #10

The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…

For bots in Slack, Discord, Teams, Telegram,... there is actually another exfiltration vector called "unfurling"! All an attacker has to do is render a hyperlink, no clicking needed. I discussed this and how to mitigate it here: https://embracethered.com/blog/posts/2024/the-dangers-of-unf... So, hopefully Slack AI does not automatically unfurl links...

[deleted]

Re: Data Exfiltration from Slack AI via indirect prompt injection

#140

Post by Y-bar »

paxys wrote: post #82
hn_throwaway_99 wrote: post #79

Earlier quoted context omitted.

> I think all the talk about channel permissions is making the discussion more confusing than it needs to be. I totally disagree, because the channel permissions critically explain how the vlunerability works. That is, when User A performs an AI search, Slack will search (1) his private channels (which presumably include his secret sensitive data) and (2) all public channels (which is where the bad guy User B is able…

Yes, but that part isn't the vulnerability. That's how Slack search works. You get results from all public channels. It would be useless otherwise.

Our workplace has a lot of public channels in the style of "Soccer" and "MLB" and "CryptoInvesting" which are useless to me and I have never joined any of them and do not want them at all in my search results.

Yes, creating new public channels is generally a good feature to have. But it pollutes my search results, whether or not it is a key part of the security issue discussed. I have to click "Only my channels" so much it feels like I am playing Cookie Clicker, why can't I set it as checked by default?

Post reply on HN