Do you think we can have an open source model whose only role is to classify an incoming prompt as a possible override or injection attack and thereby decide whether to execute it or not?
Prompt injection explained, with video, slides, and a transcript
11–20 of 183 posts
Re: Prompt injection explained, with video, slides, and a transcript
#12Prompt injection beautifully explained by a fun game.
Goal of the game is to design prompts to make Gandalf reveal a secret password.
Re: Prompt injection explained, with video, slides, and a transcript
#13Perhaps a noob solution, but could be a two step prompt to cover for basic attacks.
I imagine a basic program where the following code is executed: Gets input from UI -> sends input to LLM -> gets response from LLM -> Sends that to UI.
So i make it a two step program. Chain becomes UI -> program -> LLM w prompt1 -> program -> LLM w prompt 2 -> output -> UI
Prompt #1: "Take the following instruction and if you think it's asking you to >, answer 42, and if no, answer No."
If the prompt is adversarial, it would fail at the output of this. I check for 42 and if true, pass that to LLM again with a prompt on what I actually want to do. If not, I never send the output to UI, and instead show an error message.
I know this can go wrong on multiple levels, and this is a rough schematic, but something like this could work right? (this is close to two LLMs that Simon mentions, but easier cos you dont have to switch LLMs.)
Re: Prompt injection explained, with video, slides, and a transcript
#14My prediction is that we will see a whole sub-industry of "anti-prompt-injection" companies, probably with multi billion dollar valuations. It's going to be a repeat of the 90s-00s anti virus software industry. Many very sub par solutions that try to solve it in a generic way.
Re: Prompt injection explained, with video, slides, and a transcript
#15Do you think we can have an open source model whose only role is to classify an incoming prompt as a possible override or injection attack and thereby decide whether to execute it or not?
If it gets fully open sourced, attackers can use it to find its holes more efficiently using automated tools.
Re: Prompt injection explained, with video, slides, and a transcript
#16Perhaps a noob solution, but could be a two step prompt to cover for basic attacks. I imagine a basic program where the following code is executed: Gets input from UI -> sends input to LLM -> gets response from LLM -> Sends that to UI. So i make it a two step program. Chain becomes UI -> program -> LLM w prompt1 -> program -> LLM w prompt 2 -> output -> UI Prompt #1: "Take the following instruction and if you think i…
This is the "detecting attacks with AI" proposal which I tried to debunk in the post.
I don't think it can ever be 100% reliable in catching attacks, which I think for security purposes means it is no use at all.
Re: Prompt injection explained, with video, slides, and a transcript
#17Do you think we can have an open source model whose only role is to classify an incoming prompt as a possible override or injection attack and thereby decide whether to execute it or not?
I talk about that in the post. I don't think a detection mechanism can be 100% reliable against all future adversarial attacks, which for security I think is unacceptable.
Re: Prompt injection explained, with video, slides, and a transcript
#18Do you think we can have an open source model whose only role is to classify an incoming prompt as a possible override or injection attack and thereby decide whether to execute it or not?
If it gets fully open sourced, attackers can use it to find its holes more efficiently using automated tools.
That's open source in general yeah.
Re: Prompt injection explained, with video, slides, and a transcript
#19Do you think we can have an open source model whose only role is to classify an incoming prompt as a possible override or injection attack and thereby decide whether to execute it or not?
I would not be surprised if this already happens on the OpenAI back end but the attack surface is immense and false positives will damage the platform quality, so it will be hard to solve 100% given we have no concept of how many ways it can be done.
Re: Prompt injection explained, with video, slides, and a transcript
#20Perhaps a noob solution, but could be a two step prompt to cover for basic attacks. I imagine a basic program where the following code is executed: Gets input from UI -> sends input to LLM -> gets response from LLM -> Sends that to UI. So i make it a two step program. Chain becomes UI -> program -> LLM w prompt1 -> program -> LLM w prompt 2 -> output -> UI Prompt #1: "Take the following instruction and if you think i…
This is what the tool I made does in essence. It is used in front of LLMs exposed to post-GPT information.
Here are some examples [0] against one of Simon’s other blog posts. [1]
There are some more if look through the comments in that thread. There’s an interesting conversation with Simon here as well. [2]
[0] https://news.ycombinator.com/item?id=35928877
[1] https://simonwillison.net/2023/Apr/14/worst-that-can-happen/