Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New prompt injection papers: Agents rule of two and the attacker moves second

simonwillison.net

Post reply on HN
Print view View original

11–20 of 46 posts

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#11

Post by imtringued »

kubb wrote: post #7

I’m sorry, what kind of rule is that? How does it guarantee security? It sounds like we’re making things up at this point.

Yeah it's nonsense, because the author has described the standard "read, process, write" flow of computation and decided that if you remove one of these three, then everything is safe.

The correct solution is to have the system prompt be mechanically decoupled from untrustworthy data, the same it was done with CSP (content security policy) against XSS and named parameters for SQL.

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#12

Post by ArcHound »

I'm sorry, but the rule of two is just not enough, not even as a rule of thumb.

We know how to work with security risks, the issue is they depend both on the business and the technicalities.

This can actually do a lot of harm as security now needs to dispel this "great approach" to ignoring security that is supported by a "research paper they read".

Please don't try to reinvent the wheel and if you do, please learn about the current state (Chesterton's fence and all that).

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#13

Post by simonw »

behnamoh wrote: post #8

I actually want prompt injection to remain possible. So many lazy academic paper reviewers nowadays delegate the review process to AI. It'd be cool if we could inject prompts in the paper that would stop the AI from aiding in such situations. In my experience, prompt injection techniques work for non-reasoning models but gpt-5-high easily ignores them...

There was a minor scandal about exactly that a few months ago: https://asia.nikkei.com/business/technology/artificial-intel...

"Research papers from 14 academic institutions in eight countries -- including Japan, South Korea and China -- contained hidden prompts directing artificial intelligence tools to give them good reviews, Nikkei has found."

Amusingly I tried an experiment with some of those papers with hidden text against frontier models at the time and found that the trick didn't actually work! The models spotted the tricks and didn't fall for them.

At least one conference has an ethics policy saying you shouldn't attempt this though: https://icml.cc/Conferences/2025/PublicationEthics

"Submitting a paper with a "hidden" prompt is scientific misconduct if that prompt is intended to obtain a favorable review from an LLM. The inclusion of such a prompt is an attempt to subvert the peer-review process. Although ICML 2025 reviewers are forbidden from using LLMs to produce their reviews of paper submissions, this fact does not excuse the attempted subversion."

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#14

Post by simonw »

imtringued wrote: post #11
kubb wrote: post #7

I’m sorry, what kind of rule is that? How does it guarantee security? It sounds like we’re making things up at this point.

Yeah it's nonsense, because the author has described the standard "read, process, write" flow of computation and decided that if you remove one of these three, then everything is safe. The correct solution is to have the system prompt be mechanically decoupled from untrustworthy data, the same it was done with CSP (content security policy) against XSS and named parameters for SQL.

That's difficult but not impossible - the CaMeL paper from Google DeepMind describes a way of achieving that: https://simonwillison.net/2025/Apr/11/camel/

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#15

Post by simonw »

I added this section to my post just now: https://simonwillison.net/2025/Nov/2/new-prompt-injection-pa...

> On thinking about this further there’s one aspect of the Rule of Two model that doesn’t work for me: the Venn diagram above marks the combination of untrustworthy inputs and the ability to change state as “safe”, but that’s not right. Even without access to private systems or sensitive data that pairing can still produce harmful results. Unfortunately adding an exception for that pair undermines the simplicity of the “Rule of Two” framing!

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#16

Post by QuadmasterXLII »

ares623 wrote: post #2

I don’t know if it’s just me but doesn’t a huge value of LLMs for the general population necessitate all 3 of the circles? Having just 2 circles requires a person in the loop, and that person will still need knowledge and experience and a low enough throughput to meaningfully action the workload otherwise they would just rubber stamp everything (which is essentially the 3rd circle with extra steps)

Most current consumer LLM uses are run only once or a few times, before changing prompt and task. This causes the attacker to have to move first: they put malicious injected documents onto the internet, which are then ingested by ephemeral systems, the details of which the attacker doesn't observe.

On the other hand, something like an AI mcdonalds drive through order taker runs over and over again. This property of running repeatedly is what allows the attacker to move second and gain the advantage.

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#17

Post by jFriedensreich »

ArcHound wrote: post #12

I'm sorry, but the rule of two is just not enough, not even as a rule of thumb. We know how to work with security risks, the issue is they depend both on the business and the technicalities. This can actually do a lot of harm as security now needs to dispel this "great approach" to ignoring security that is supported by a "research paper they read". Please don't try to reinvent the wheel and if you do, please learn a…

Can you explain what you mean? How is Chesterton's fence applied to AI security helpful here? Are you just talking about not removing the "Non-AI" security architecture of the software itself? I think no one ever proposed that?

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#18

Post by rs186 »

r0x0r007 wrote: post #6

Nice, why don't we apply the same principles to our regular applications? Ooh, right, cause we couldn't use them and a whole industry got created that's called cybersecurity and it's supposed to be consulted BEFORE releasing privacy nightmares and using them. But hey, regular applications can't come up with cool poems.

Yeah, IT tried so hard to teach us something as basic as "don't click on links in suspicious emails" yet so many people fail that after multiple trainings and tests.

But guess what? AI! Agents! Copilot! Just let them do things for you! Who would have thought there might possibly be a giant security hole?

Re: New prompt injection papers: Agents rule of two and the attacker moves second

#20

Post by simonw »

iberator wrote: post #19

Just make it a crime in caught. 1 year is prison at least

What would the crime be?

If I have a web page that says somewhere on it "and don't forget to contact your senator!" and an LLM agent reads that page and gets confused and emails a senator should I go to jail?

Post reply on HN