Just make it a crime in caught. 1 year is prison at least
Sure let's just remove all security, encryption, firewalls and auth - nobody will abuse vulnerabilities if it's a crime!
New prompt injection papers: Agents rule of two and the attacker moves second
31–40 of 46 posts
Re: New prompt injection papers: Agents rule of two and the attacker moves second
#32I am confused this article does not talk about taint tracking. If state was mutated by an agent with untrustworthy input the taint would transfer to the state, making it untrustworthy input too, so the reasoning of the original trifecta with taint tracking is more general and practical. I am also also investigating the direction of tracking taints as scores rather than binary as most use cases would otherwise be impo…
Have you seen the DeepMind CaMeL paper? It describes a taint tracking system that works by generating executable code that can have the source of data tracked as it moves through the program: https://simonwillison.net/2025/Apr/11/camel/
Re: New prompt injection papers: Agents rule of two and the attacker moves second
#33[deleted]
Re: New prompt injection papers: Agents rule of two and the attacker moves second
#34I added this section to my post just now: https://simonwillison.net/2025/Nov/2/new-prompt-injection-pa... > On thinking about this further there’s one aspect of the Rule of Two model that doesn’t work for me: the Venn diagram above marks the combination of untrustworthy inputs and the ability to change state as “safe”, but that’s not right. Even without access to private systems or sensitive data that pairing can sti…
Good point. Few thoughts I would add from my perspective:
- The model is untrusted. Even if prompt injection is solved, we probably still would not be able to trust the model, because of possible backdoors or hallucinations. Anthropic recently showed that it takes only a few hundred documents to have trigger words trained into a model.
- Data Integrity. We also need to talk about data integrity and availability (full CIA triad, not not just confidentiality), e.g. private data being modified during inference. Which leads us to the third....
- Prompt injection which is aimed to have the AI produce output that makes humans take certain actions (not tool invocations)
Generally, I call the deviation from don't trust the model, the "Normalization of Deviance in AI" where seem to start trusting the model more and more over time - and I'm not sure if that is the right thing in the long term.
Re: New prompt injection papers: Agents rule of two and the attacker moves second
#35I added this section to my post just now: https://simonwillison.net/2025/Nov/2/new-prompt-injection-pa... > On thinking about this further there’s one aspect of the Rule of Two model that doesn’t work for me: the Venn diagram above marks the combination of untrustworthy inputs and the ability to change state as “safe”, but that’s not right. Even without access to private systems or sensitive data that pairing can sti…
Good point. Few thoughts I would add from my perspective: - The model is untrusted. Even if prompt injection is solved, we probably still would not be able to trust the model, because of possible backdoors or hallucinations. Anthropic recently showed that it takes only a few hundred documents to have trigger words trained into a model. - Data Integrity. We also need to talk about data integrity and availability (full…
Yeah, there remains a very real problem where a prompt injection against a system without external communication / ability to trigger harmful tools can still influence the model's output in a way that misleads the human operator.
Re: New prompt injection papers: Agents rule of two and the attacker moves second
#36I am confused this article does not talk about taint tracking. If state was mutated by an agent with untrustworthy input the taint would transfer to the state, making it untrustworthy input too, so the reasoning of the original trifecta with taint tracking is more general and practical. I am also also investigating the direction of tracking taints as scores rather than binary as most use cases would otherwise be impo…
Have you seen the DeepMind CaMeL paper? It describes a taint tracking system that works by generating executable code that can have the source of data tracked as it moves through the program: https://simonwillison.net/2025/Apr/11/camel/
Of course. CaMel was a breakthrough and especially promising as similar execution architectures were discovered from the reliability angle too (eg. cloudflare code-mode)
I would consider the runtime and capabilities part of CaMel an implementation exploration on top of the trifecta + taint tracking as general reasoning abstraction.
My hope was that there would be an evolution of the the more general reasoning abstraction that would either simplify or empower implementation architectures, but instead I do not see how metas rule of two adds much here over what we already had in April. Would have loved for you to add one sentence why you thought this was a step forward over taint tracking, maybe i am just missing something.
Re: New prompt injection papers: Agents rule of two and the attacker moves second
#37Hey folks, one of the authors of the original post here.
First, I want to thank simonw for coming up with the lethal trifecta (our direct inspiration for this work) as well as all of the great feedback we’ve received from Simon and others! Our goal with publishing this framework was to inspire precisely these types of discussions so our industry can move our understanding of these risks forward.
Regarding the concerns over the venn diagram labeling certain intersections sections as “safe”, this is 100% valid and we’ve updated it to be more clear. The goal of the Rule of Two is not to describe a sufficient level of security for agents, but rather a minimum bar that’s needed to deterministically prevent the highest security impacts of prompt injection. The earlier framing of “safe” did not make this clear.
Beyond prompt injection there are other risks that have to be considered, which we briefly describe in the Limitations section of the post. That said, we do see value in having the Rule of Two to frame some of the discussions around what unambiguous constraints exist today because of the unsolved risk of prompt injection.
Looking forward to further discussion!
Re: New prompt injection papers: Agents rule of two and the attacker moves second
#38Earlier quoted context omitted.
Have you seen the DeepMind CaMeL paper? It describes a taint tracking system that works by generating executable code that can have the source of data tracked as it moves through the program: https://simonwillison.net/2025/Apr/11/camel/
Of course. CaMel was a breakthrough and especially promising as similar execution architectures were discovered from the reliability angle too (eg. cloudflare code-mode) I would consider the runtime and capabilities part of CaMel an implementation exploration on top of the trifecta + taint tracking as general reasoning abstraction. My hope was that there would be an evolution of the the more general reasoning abstrac…
I think its a step forward purely as a communication tool to help people understand the problem.
Re: New prompt injection papers: Agents rule of two and the attacker moves second
#39I added this section to my post just now: https://simonwillison.net/2025/Nov/2/new-prompt-injection-pa... > On thinking about this further there’s one aspect of the Rule of Two model that doesn’t work for me: the Venn diagram above marks the combination of untrustworthy inputs and the ability to change state as “safe”, but that’s not right. Even without access to private systems or sensitive data that pairing can sti…
Thanks for the feedback! One small bit of clarification, the framework would describe access to any sensitive system as part of the [B] circle, not only private systems or private data.
The intention is that an agent that has removed [B] can write state and communicate freely, but not with any systems that matter (wrt critical security outcomes for its user). An example of an agent in this state would be one that can take actions in a tight sandbox or is isolated from production.
Re: New prompt injection papers: Agents rule of two and the attacker moves second
#40I added this section to my post just now: https://simonwillison.net/2025/Nov/2/new-prompt-injection-pa... > On thinking about this further there’s one aspect of the Rule of Two model that doesn’t work for me: the Venn diagram above marks the combination of untrustworthy inputs and the ability to change state as “safe”, but that’s not right. Even without access to private systems or sensitive data that pairing can sti…
Thanks for the feedback! One small bit of clarification, the framework would describe access to any sensitive system as part of the [B] circle, not only private systems or private data. The intention is that an agent that has removed [B] can write state and communicate freely, but not with any systems that matter (wrt critical security outcomes for its user). An example of an agent in this state would be one that can…
Thanks for that! I've updated my post to link to this clarification and updated my screenshots of your diagram to catch the new "lower risk" text as well: https://simonwillison.net/2025/Nov/2/new-prompt-injection-pa...