Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New prompt injection papers: Agents rule of two and the attacker moves second

simonwillison.net

Post reply on HN
Print view View original

21–30 of 46 posts

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#21

Post by ArcHound »

jFriedensreich wrote: post #17
ArcHound wrote: post #12

I'm sorry, but the rule of two is just not enough, not even as a rule of thumb. We know how to work with security risks, the issue is they depend both on the business and the technicalities. This can actually do a lot of harm as security now needs to dispel this "great approach" to ignoring security that is supported by a "research paper they read". Please don't try to reinvent the wheel and if you do, please learn a…

Can you explain what you mean? How is Chesterton's fence applied to AI security helpful here? Are you just talking about not removing the "Non-AI" security architecture of the software itself? I think no one ever proposed that?

Right, what got me going is the reduction of plenty cyber security concepts into a simple "safe" label in the diagram.

So what I meant is that before you discard all of the current security practices, it's better to learn about the current approach.

From another angle, maybe the diagram could be fixed with changing "safe" to "danger" and "danger" to "OMG stop". But that also discards the business perspective and the nature of the protected asset.

I am also happy to see the edit in the article, props to the author for that!

And to address the last question, no one proposed that right now, yes. But I was in plenty of discussions about security approaches. And let me tell you, sometimes it only takes one sentence that the leadership likes to hear to detail the whole approach (especially if it results in cost savings). So I might be extra sensitive to such ideas and I try to uproot them before they bloom fully.

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#22

Post by ArcHound »

simonw wrote: post #15

I added this section to my post just now: https://simonwillison.net/2025/Nov/2/new-prompt-injection-pa... > On thinking about this further there’s one aspect of the Rule of Two model that doesn’t work for me: the Venn diagram above marks the combination of untrustworthy inputs and the ability to change state as “safe”, but that’s not right. Even without access to private systems or sensitive data that pairing can sti…

I love to see this. As much as we try for simple security principles, the damn things have a way to become complicated quickly.

Perhaps the diagram highlights the common risky parts of these apps and we gain more risk as we keep increasing the scope? Maybe we can do some handovers and protocols to separate these concerns?

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#23

Post by kloud »

simonw wrote: post #15

I added this section to my post just now: https://simonwillison.net/2025/Nov/2/new-prompt-injection-pa... > On thinking about this further there’s one aspect of the Rule of Two model that doesn’t work for me: the Venn diagram above marks the combination of untrustworthy inputs and the ability to change state as “safe”, but that’s not right. Even without access to private systems or sensitive data that pairing can sti…

Also in the context of LLMs I think model weights themselves could be considered an untrusted input, because who knows what was in the training dataset. Even an innocent looking prompt could potentially trigger a harmful outcome.

In that regard it reminds me of the CAP theorem, which also has three parts. However, in practice partitioning in distributed systems is given, so the choice is just between availability or consistency.

So in the case of lethal trifecta it is either private data or external communication, but the leg between these two will always have some risk.

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#24

Post by jFriedensreich »

I am confused this article does not talk about taint tracking. If state was mutated by an agent with untrustworthy input the taint would transfer to the state, making it untrustworthy input too, so the reasoning of the original trifecta with taint tracking is more general and practical. I am also also investigating the direction of tracking taints as scores rather than binary as most use cases would otherwise be impossible to do at all autonomous. Eg. with sensitivity scores to data, trust scores to inputs (that can be improved by eg. human review). One important limit that needs way more research is how to transfer the minimal needed information from a tainted context into an untainted fresh context without transferring all the taints. The only solution i currently have is by compaction and human review, if possible aided with schema enforcement and optimised UI for the use case. This unfortunately cannot solve encoded information that humans cannot see, but it seems that issue will never be solvable outside alignment research.

PS: An example how scores are helpful: Using browser tab titles in the context would by definition have the worst trust score possible. But truncating titles to only the user-visible parts could lower this to acceptable for autonomous execution if the data was just mildly sensitive.

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#25

Post by jFriedensreich »

ArcHound wrote: post #21
jFriedensreich wrote: post #17

Earlier quoted context omitted.

Can you explain what you mean? How is Chesterton's fence applied to AI security helpful here? Are you just talking about not removing the "Non-AI" security architecture of the software itself? I think no one ever proposed that?

Right, what got me going is the reduction of plenty cyber security concepts into a simple "safe" label in the diagram. So what I meant is that before you discard all of the current security practices, it's better to learn about the current approach. From another angle, maybe the diagram could be fixed with changing "safe" to "danger" and "danger" to "OMG stop". But that also discards the business perspective and the…

Hmm, what do you mean by current approach? This is new territory and agent safety is an unsolved problem, there is no current approach, except you mean not doing agent systems and using humans. The trifecta is just a tool on the level of physics saying "ignore friction", we assume the model itself is trustworthy and not poisoned most of the time too, but of course when designing a real world system you need to factor that in too.

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#26

Post by ArcHound »

jFriedensreich wrote: post #25
ArcHound wrote: post #21

Earlier quoted context omitted.

Right, what got me going is the reduction of plenty cyber security concepts into a simple "safe" label in the diagram. So what I meant is that before you discard all of the current security practices, it's better to learn about the current approach. From another angle, maybe the diagram could be fixed with changing "safe" to "danger" and "danger" to "OMG stop". But that also discards the business perspective and the…

Hmm, what do you mean by current approach? This is new territory and agent safety is an unsolved problem, there is no current approach, except you mean not doing agent systems and using humans. The trifecta is just a tool on the level of physics saying "ignore friction", we assume the model itself is trustworthy and not poisoned most of the time too, but of course when designing a real world system you need to factor…

Yes, by current approach I mean security best practices for non-LLM apps. Plenty of those are directly applicable.

And yes, LLMs have some challenges. But discarding all of the lessons and principles we've discovered over the years is not the way. And if we need to discard some of them, we should understand exactly why they are no longer applicable.

EDIT: I know that models need to omit stuff to be useful. But this model omits too much - claiming that something is "safe" should be a red flag to all security workers.

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#27

Post by gs17 »

> [A] An agent can process untrustworthy inputs

> [B] An agent can have access to sensitive systems or private data

> [C] An agent can change state or communicate externally

Somewhat reminds me of the CAP theorem, where you can pick two of three, but one is effectively required for something useful. It seems more like the choice is really between "untrustworthy inputs" and "sensitive systems", which makes sense.

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#28

Post by causal »

simonw wrote: post #15

I added this section to my post just now: https://simonwillison.net/2025/Nov/2/new-prompt-injection-pa... > On thinking about this further there’s one aspect of the Rule of Two model that doesn’t work for me: the Venn diagram above marks the combination of untrustworthy inputs and the ability to change state as “safe”, but that’s not right. Even without access to private systems or sensitive data that pairing can sti…

I think the rule of 2 would work if it kept the 3 from your lethal trifecta. "Change state" should be not be paired with "communicate externally".

And even then that's just to avoid data exfiltration- if you can't communicate externally but can change state, damage can still be done.

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#29

Post by causal »

jFriedensreich wrote: post #24

I am confused this article does not talk about taint tracking. If state was mutated by an agent with untrustworthy input the taint would transfer to the state, making it untrustworthy input too, so the reasoning of the original trifecta with taint tracking is more general and practical. I am also also investigating the direction of tracking taints as scores rather than binary as most use cases would otherwise be impo…

Totally. I think the original "Lethal trifecta" post by OP only pertained to data exfiltration and never included changing state (maybe was implied by sensitive data access).

Rule of 2 model has holes.

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#30

Post by cubefox »

simonw wrote: post #13
behnamoh wrote: post #8

I actually want prompt injection to remain possible. So many lazy academic paper reviewers nowadays delegate the review process to AI. It'd be cool if we could inject prompts in the paper that would stop the AI from aiding in such situations. In my experience, prompt injection techniques work for non-reasoning models but gpt-5-high easily ignores them...

There was a minor scandal about exactly that a few months ago: https://asia.nikkei.com/business/technology/artificial-intel... "Research papers from 14 academic institutions in eight countries -- including Japan, South Korea and China -- contained hidden prompts directing artificial intelligence tools to give them good reviews, Nikkei has found." Amusingly I tried an experiment with some of those papers with hidden t…

Intuitively it does excuse it though.
Post reply on HN