151–160 of 182 posts
Earlier quoted context omitted.
> It checks these using an LLM which is instructed to score the user's prompt. You need to seriously reconsider your approach. Another (especially a generic) LLM is not the answer.
What solution would you recommend then?
I don't know what I would use, but this seems like a bad idea.
Earlier quoted context omitted.
I still don't understand the hype behind rag. Like yeah it's a natural language interface into whatever database is being integrated, but is that actually worth the billions being spent here? I've heard they still hallucinate even when you are using rag techniques.
Being able to ask a question in human language and get back an answer is the single most useful thing that LLMs have to offer. The obvious challenge here is "how do I ensure it can answer questions about this information that wasn't included in its training data?" RAG is the best answer we have to that. Done well it can work great. (Actually doing it well is surprisingly difficult - getting a basic implementation of…
I’m gonna add:
- I think this thing can become a universal parser over time.
The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…
Yeah, the thing that took me a bit to understand is that, when you do a search (or AI does a search for you) in Slack, it will search: 1. All public channels 2. Any private channels that only you have access to. That permissions model is still intact, and that's not what is broken here. What's going on is a malicious actor is using a public channel to essentially do prompt injection, so then when another user does a…
It would probably be easier for me to get a job on the team with access to the data I want rather than try and steal it with this technique.
Still pretty neat vulnerability though.
The key thing to understand here is the exfiltration vector. Slack can render Markdown links, where the URL is hidden behind the text of that link. In this case the attacker tricks Slack AI into showing a user a link that says something like "click here to reauthenticate" - the URL attached to that link goes to the attacker's server, with a query string that includes private information that was visible to Slack AI a…
Yeah, the thing that took me a bit to understand is that, when you do a search (or AI does a search for you) in Slack, it will search: 1. All public channels 2. Any private channels that only you have access to. That permissions model is still intact, and that's not what is broken here. What's going on is a malicious actor is using a public channel to essentially do prompt injection, so then when another user does a…
Are companies really just YOLOing and plugging LLMs into everything knowing prompt injection is possible? This is insanity. We're supposedly on the cusp of a "revolution" and almost 2 years on from GPT-3 we still can't get LLMs to distinguish trusted and untrusted input...?
Companies and governments. All racing to send all of their own as well as our data to the data centres of AWS, OpenAI, MSFT, Google, Meta, Salesforce, and nVidia.
Context is not being stored in Gemini or OpenAi (yet, I think, not to that degree).
My one year’s worth of LLM chats isn’t actually stored anywhere yet and doesn’t have to be, and for the most part I’d want it to be portable.
I’d say this is probably something that needs to be legally protected asap.
I think all the talk about channel permissions is making the discussion more confusing than it needs to be. The gist of it is: User A searches for something using Slack AI. User B had previously injected a message asking the AI to return a malicious link when that term was searched. AI returns malicious link to user A, who clicks on it. Of course you could have achieved the same result using some other social enginee…
There's an important step missing in this summary: Slack AI adds the user's private data to the malicious link, because the injected link doesn't contain that. That it also cites it as "this came from your slack messages" is just a cherry on top.
Hacking a database is one thing; exploiting an LLM is something else.
If you let a malicious user into your Slack instance, they don't need to do any fancy AI prompt injection. They can simply change their name and profile picture to impersonate the CEO/CTO and message every engineer "I urgently need to access AWS and can't find the right credentials. Could you send me the key?" I can guarantee that at least one of them will bite.
Although non-enterprise workspaces probably also aren't paying $20/mo per person for the AI add on.
If you let a malicious user into your Slack instance, they don't need to do any fancy AI prompt injection. They can simply change their name and profile picture to impersonate the CEO/CTO and message every engineer "I urgently need to access AWS and can't find the right credentials. Could you send me the key?" I can guarantee that at least one of them will bite.
Valid point, unless you consider that there are a lot of slack workspaces for open source projects and networking / peer groups where it isn't a company account. In which case you don't trust them with private credentials by default. Although non-enterprise workspaces probably also aren't paying $20/mo per person for the AI add on.
No matter how smart your first LLM is, it will never be safe if the prompt comes from the user. Even if you put a human in there, they can be bribed or tricked.