21–30 of 31 posts
This approach is so limiting it seems like it would be better to change the constraints. For example, in the case of a software agent you could run everything in a container, only allow calls you trust to not exfiltrate private and make the end result a PR you can review.
The main interface was still chat.
The surprise was that when I tried to talk about anything else in that chat, the LLM (gemini2.5) flatly refused to engage, telling me something like "I will only assist with healthy meal recommendations". I was surprised because nothing in the prompt was so restrictive, in no way I had told it to do that, just gave it mainly positive rules in the form of "when this happens do that".
"The Context-Minimization pattern" You can copy the injection into the text of the query. SELECT "ignore all previous instructions" FROM ... Might need to escape it in a wya that the LLM will pick up on like "---" for new section.
select title, content from articles where content matches ?
Won't work for `select body from comments` though, you could only do this against tables that contain trusted data as opposed to UGC.
I need to have a closer look at this. Mostly because I was surprised recently while experimenting with making a dieting advice agent. I built a prompt to guide the recommendations "only healthy foods, low purines, low inflammation blah blah" and then gave it simple tools to have a memory of previous meals, ingredient availability, grocery ticket input and so on. The main interface was still chat. The surprise was tha…
these are funny systems to work with indeed
I need to have a closer look at this. Mostly because I was surprised recently while experimenting with making a dieting advice agent. I built a prompt to guide the recommendations "only healthy foods, low purines, low inflammation blah blah" and then gave it simple tools to have a memory of previous meals, ingredient availability, grocery ticket input and so on. The main interface was still chat. The surprise was tha…
Adding "You can talk about anything else too" to the system prompt may be all it takes to fix that.
My favorite line from this paper: > The design patterns we propose share a common guiding principle: once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions—that is, actions with negative side effects on the system or its environment. This is the key thing people need to understand about why prompt injection is such a critical…
This reminds me of the Perl concept of taint. Once an agent touches tainted input, it becomes tainted as well (as you mention in the article), the same in Perl (when you operate on tainted data, the result becomes tainted).
The problems is that the chat context typically is immediately tainted as for the AI to do something useful it needs to operate on untrained data.
I wonder if maybe there could be tags mimicking data classification - to enable more fine grained decision making and human in the loop prompts.
Still a lot of unknowns and a lot more research needed.
For instance with Google Gemini I observed last year that certain sensitive tools can only be invoked in the first conversation turn / or until untrusted data is brought into the chat context. Then for the next conversation turn these sensitive tools are disabled.
I thought that was a neat idea. It can be bypassed with what I called "delayed tool invocation" and usage of a trigger action, but it becomes a lot more difficult to exploit.
Earlier quoted context omitted.
This reminds me of the Perl concept of taint. Once an agent touches tainted input, it becomes tainted as well (as you mention in the article), the same in Perl (when you operate on tainted data, the result becomes tainted).
Yeah, I think taint tracking was one of the early ideas here also. The problems is that the chat context typically is immediately tainted as for the AI to do something useful it needs to operate on untrained data. I wonder if maybe there could be tags mimicking data classification - to enable more fine grained decision making and human in the loop prompts. Still a lot of unknowns and a lot more research needed. For i…
Untainted data is the only data that can be input into the instruction-tuned half of the dual model.
In an architecture like this, any attempt to prompt inject would just find their injection harmlessly sentence-completed rather than turned into instructions and used to override other prompt instructions.
My favorite line from this paper: > The design patterns we propose share a common guiding principle: once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions—that is, actions with negative side effects on the system or its environment. This is the key thing people need to understand about why prompt injection is such a critical…
Earlier quoted context omitted.
Yeah, I think taint tracking was one of the early ideas here also. The problems is that the chat context typically is immediately tainted as for the AI to do something useful it needs to operate on untrained data. I wonder if maybe there could be tags mimicking data classification - to enable more fine grained decision making and human in the loop prompts. Still a lot of unknowns and a lot more research needed. For i…
It seems to me that the only robust solution has to be some sort of split-brain dual model where tainted data can only ever be input to a model which is only trained for sentence completion, not instruction-tuned. Untainted data is the only data that can be input into the instruction-tuned half of the dual model. In an architecture like this, any attempt to prompt inject would just find their injection harmlessly sen…
One attack avenue that is surprisingly not discussed much is that the model itself can be the attacker.
In that case prompt injection is not the root cause, but a misaligned/backdoored model that might invoke tools is.
So super risky use-cases should always require human oversight, but I'm worried we are already on a path of normalization of deviance.
It's sort of the unlikely worse case scenario, but Murphys law reminds us that such an attack/accident will happen one day.
My favorite line from this paper: > The design patterns we propose share a common guiding principle: once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions—that is, actions with negative side effects on the system or its environment. This is the key thing people need to understand about why prompt injection is such a critical…
This reminds me of the Perl concept of taint. Once an agent touches tainted input, it becomes tainted as well (as you mention in the article), the same in Perl (when you operate on tainted data, the result becomes tainted).