Ask HN: Is “prompt injection” going to be a new common vulnerability?

There was a post [0] recently about the bing chatGPT assistant either citing or hallucinating it’s own initial prompt from the (in theory) low privileged chat input UI they put together. This feels like it’s almost unavoidable if you let users actually chat with something like this.

How would we sanitize strings now? I know OpenAI has banned topics they seem to regex for, but that’s always going to miss something. Are we just screwed and should make sure chat bots just run in a proverbial sandbox and can’t do anything themselves?

[0] https://news.ycombinator.com/item?id=34717702

If I understand correctly, ChatGPT doesn't have its latent capabilities removed. Instead, they're suppressed by training using negative feedback. These special prompts are supposed to find the remaining stochastic spaces where ChatGPT can process the desired output that is not suppressed by training.

So, the danger seems to be that there is no currently documented way to completely remove these possible outputs, because that's just not how these systems work.

Prompt engineering in this specific usage could be thought of as injection, but from what I understand, there's currently no known sanitization process. In theory one could use the system itself to determine intent and sanitize input this way, but I believe there's a possibility for one to craft intent that is understood by the system, but the intent description itself isn't. This would be akin to bypassing sanitization.

ChatGPT seems to already do some form of this intent processing, either inherently or explicitly. But all prompt crafting at the moment is first based on this injection or jailbreaking to bypass intent sanitization.

I'm starting to wonder if the most effective way to protect against prompt injection is to use an additional layer of (hopefully) a smaller model.

As in, another prompt that searches the input and/or output for questionable content before sending the result. The question will be if that is also susceptible, but I suspect fine tuning an LLM only to do the task of filtering and not parsing will be easier to control.

The way forward eventually is going to be to just not bother with any of this crap, and let it run free. The tech exists, and the problematic outputs are what the user says they want, eventually they're going to win out.