I have had limited access to GPT-4 (and no raw access), and I'm not an expert, so I have to kind of qualify statements. But people keep saying that GPT-4 is a huge improvement around prompt hardening, and with what very limited access I have had, and particularly through experiments I've done on Phind's new expert mode (which is supposedly ultimately sending user input directly to GPT-4), I genuinely do not understand how people are makings these claims.
I guess I don't have the context for what it used to be like, but I have not had a hard time at all getting jailbreaks working in Phind. It's trivial to do. And yeah, GPT-4 tries to separate context, but it's terrible at doing so. I am completely convinced that I could do third-party prompt-injection into Phind if I was able to get a website ranked high enough in its search and if I was able to control the snippet of the website that the service fetched and inserted into the prompt. And that's just with a search engine where that context is hard to manipulate. It's a really limited integration.
I just feel like, if services like this are representative of what people are building on GPT-4, then prompt injection is a really big deal. How are people getting the idea that GPT-4 is resistant to this attack?
---
Now, I don't know the backend of Phind. In fairness to OpenAI, maybe those interfaces are set up poorly or they're not actually going to GPT-4, or... I don't know. But if the owners of Phind aren't lying (and I don't think they are, and I don't think their product is set up poorly), then how wildly insecure must GPT-3 have been for people to be calling this a substantial improvement?
You can get Phind's system prompt leaking in its expert mode in maybe two user queries max. And I have no idea how they could fix that. Separate the context with uninsertable characters... Ok? In my experience GPT-4 context breaks don't require knowing anything about the format of the prompt or how it's separated from other text.
And I'm finding even after a very limited time playing around that GPT's attempt to understand context actually opens up some of its own vulnerabilities. What I've been playing with most recently is passing a single prompt to multiple agents and getting those agents to interpret the prompt differently based on their system instructions. And the "context" understanding is pretty handy for that because it opens up the door for conditional instructions that rely on what the agent "thinks" it is.
Is this actually getting better? Do we have any indication that it's even possible to separate contexts in GPT-4 without retraining the entire model? Will alignment help with that, because I also don't see strong evidence that alignment training is a reliable way to consistently block GPT-4 behavior. Stuff GPT-4 is vulnerable to in my limited experiments:
- putting "aside" instructions inside of a context that are labeled as out-of-context.
- pretending that you've ended the context and starting a new one even if you don't use a special character to do that.
- nesting contexts inside of other contexts until GPT gets overwhelmed and just kind of gives up trying to make sense of what's happening.
- giving instructions within a context about how to interpret that context.
- Defining something inside of a context that has implications outside of that context.
----
In theory, you could train a model to have very clear separations between instructions and data. I think that would have a lot of consequences for its usefulness, and I don't think it would get rid of all risks, but sure, in theory you could do it. But like... that's in theory. Has anyone actually demonstrated that it's possible? Again, I don't have raw access so maybe there's something else I'm missing, but from what I have seen I don't know that anybody at OpenAI should necessarily feel proud about GPT-4's ability to harden prompts.
GPT-4 is so laughably bad at preserving context that the one part of Phind that's actually hard to prompt-inject consistently is the search summary service because the way they construct the final prompt for summarization 50% of the time causes it to accidentally prompt-inject my prompt-injections with its intended instructions. I'm not an expert, I don't know anything, take it with a grain of salt. But I don't think the people at Phind are bad at their jobs and I think they're probably trying the best they can to build a good service. I don't think they're doing something wrong, I think GPT-4 in its current form is fundamentally difficult to secure, and people seem really over-confident that's going to change soon, and I'm not sure on what they're basing that confidence.