Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Exfiltration from Slack AI via indirect prompt injection

promptarmor.substack.com

Post reply on HN
Print view View original

61–70 of 182 posts

Re: Data Exfiltration from Slack AI via indirect prompt injection

#61

Post by baxtr »

benreesman wrote: post #59
baxtr wrote: post #56

Earlier quoted context omitted.

Really? Have you looked into the Marriott data beach case?

This one? “Marriott finds financial reprieve in reduced GDPR penalty” [1]? They seem to have been whacked several times without a C-Suite Exec missing a ski-vacation. If I’m ignorant please correct me but I’m unaware of anyone important at Marriott choosing an E-Class rather than an S-Class over it. [1] https://www.cybersecuritydive.com/news/marriott-finds-financ...

Nah, European GDPR fines are a joke.

I’m talking about the US class action. The sum I read about is in the billions.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#62

Post by lesuorac »

baxtr wrote: post #56
benreesman wrote: post #54

Earlier quoted context omitted.

I think the key thing to understand is that there are never. Full Stop. Any meaningful consequences to getting pwned on user data. Every big tech company has a blanket, unassailable pass on blowing it now.

Really? Have you looked into the Marriott data beach case?

Not really. Quick search just seems like the only notable thing is that it's allowed to be a class action.

But how consequential can it be if it doesn't event get more than a passing mention of the wikipedia page. [1]

[1]: https://en.wikipedia.org/wiki/Marriott_International#Marriot...

Re: Data Exfiltration from Slack AI via indirect prompt injection

#63

Post by benreesman »

baxtr wrote: post #61
benreesman wrote: post #59

Earlier quoted context omitted.

This one? “Marriott finds financial reprieve in reduced GDPR penalty” [1]? They seem to have been whacked several times without a C-Suite Exec missing a ski-vacation. If I’m ignorant please correct me but I’m unaware of anyone important at Marriott choosing an E-Class rather than an S-Class over it. [1] https://www.cybersecuritydive.com/news/marriott-finds-financ...

Nah, European GDPR fines are a joke. I’m talking about the US class action. The sum I read about is in the billions.

It sounds like I might be full of it, would you kindly link me to a source?

Re: Data Exfiltration from Slack AI via indirect prompt injection

#65

Post by saintfire »

Groxx wrote: post #14

> The victim does not have to be in the public channel for the attack to work Oh boy this is gonna be good. > Note also that the citation [1] does not refer to the attacker’s channel. Rather, it only refers to the private channel that the user put their API key in. This is in violation of the correct citation behavior, which is that every message which contributed to an answer should be cited. I really don't understa…

I do find citations helpful because I can check if the LLM just hallucinated.

It's not that seeing a citation makes me trust it, it's that I can fact check it.

Kagi's FastGPT is the first LLM I've enjoyed using because I can treat it as a summary of sources and then confirm at a primary source. Rather than sifting through increasingly irrelevant sources that pollute the internet.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#66

Post by cj »

Groxx wrote: post #14

> The victim does not have to be in the public channel for the attack to work Oh boy this is gonna be good. > Note also that the citation [1] does not refer to the attacker’s channel. Rather, it only refers to the private channel that the user put their API key in. This is in violation of the correct citation behavior, which is that every message which contributed to an answer should be cited. I really don't understa…

> I really don't understand why anyone expects LLM citations to be correct

It can be done if you do something like:

1. Take user’s prompt, ask LLM to convert the prompt into a elastic search query (for example)

2. Use elastic search (or similar) to find sources that contain the keywords

3. Ask LLM to limit its response to information on that page

4. Insert the citations based on step 2 which you know are real sources

Or at least that’s my naive way of how I would design it.

The key is limiting the LLM’s knowledge to information in the source. Then the only real concern is hallucination and the value of the information surfaced by Elastic Search

I realize this approach also ignores benefits (maybe?) of allowing it full reign on the entire corpus of information, though.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#69

Post by Jenk »

samstave wrote: post #48
digging wrote: post #23

Earlier quoted context omitted.

> It’s like everyone lost their collective mind and forgot the lessons of the past twenty years. I think this has it backwards, and actually applies to every safety and security procedure in any field. Only the experts ever cared about or learned the lessons. The CEOs never learned anything about security; it's someone else's problem. So there was nothing for AI peddlers to forget, they just found a gap in the armor…

Some ( all ) CEOs learned that offering a free month coupon/voucher for Future Security Services to secure your information against a breach like the one that just happened on the platform that's offering you a free voucher to secure your data that sits on the platform that was compromised and leaked your data, is a nifty-clean way to handle such legal inconveniences. Oh, and some supposed financial penalty is claime…

Techies aren't immune either, before we all follow the "blame management" bandwagon for the 2^101-tieth time.

CEOs aren't the reason supply chain attacks are absolutely rife with problems right now. That's entirely on the technical experts who created all of those pinnacle achievements in tech ranging from tech-led orgs and open source community built package ecosystems. Arbitrary code execution in homebrew, scoop, chocolatey, npm, expo, cocoapods, pip... you name it, it's got infected.

The LastPass data breach happened because _the_ alpha-geek in that building got sloppy and kept the keys to prod on their laptop _and_ got phised.

Re: Data Exfiltration from Slack AI via indirect prompt injection

#70

Post by surfingdino »

cedws wrote: post #36

Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?

Companies and governments. All racing to send all of their own as well as our data to the data centres of AWS, OpenAI, MSFT, Google, Meta, Salesforce, and nVidia.
Post reply on HN