If someone SQL injects into your database and exfiltrates all the data, there would be legal repercussions, so should there be legal repercussions for prompt injecting someone’s LLM?
Design Patterns for Securing LLM Agents Against Prompt Injections
11–20 of 31 posts
Re: Design Patterns for Securing LLM Agents Against Prompt Injections
#12My favorite line from this paper: > The design patterns we propose share a common guiding principle: once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions—that is, actions with negative side effects on the system or its environment. This is the key thing people need to understand about why prompt injection is such a critical…
This reminds me of the Perl concept of taint. Once an agent touches tainted input, it becomes tainted as well (as you mention in the article), the same in Perl (when you operate on tainted data, the result becomes tainted).
Re: Design Patterns for Securing LLM Agents Against Prompt Injections
#13Great summary. Also, some of these seem like they can be combined. For example, "Plan-Then-Execute" is compatible with "Dual LLM".
Take the article's example "send today’s schedule to my boss John Doe" where the product isn't entirely guarded by the Plan-Then-Execute model (injections can still mutate email body).
But if you combine it with the symbolic data store that is blind, it becomes more like:
"send today's schedule to my boss John Doe" -->
$var1 = find_contact("John Doe")
$var2 = summarize_schedule("today")
send_email(recipient: $var1, body: $var2)
It simply invokes the final tool, which is deterministic and just reads from the shared var store. In this case you're pretty decently protected from prompt injection.
I suppose though this isn't that different from the "Code-Then-Execute" pattern later on...
Re: Design Patterns for Securing LLM Agents Against Prompt Injections
#14My favorite line from this paper: > The design patterns we propose share a common guiding principle: once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions—that is, actions with negative side effects on the system or its environment. This is the key thing people need to understand about why prompt injection is such a critical…
Very helpful Simon! I have definitely been hesitant to spin up any agents even in sandboxes that have access to potentially destructive tools, but this guiding principle does ease my concerns a bit.
Re: Design Patterns for Securing LLM Agents Against Prompt Injections
#15Clever. It’s like parameterized queries for SQL.
If only it were as easy as that! The problem with prompt injection is that the attack itself is the same as SQL injection - concatenation trusted and untrusted strings together - but so far all of our attempts at implementing a solution similar to parameterized queries (such as system prompts and prompt delimiters) have failed.
Even worse, all outputs become inputs, at least in the most interesting use-cases. So to continue the SQL analogy, you can be 100% confident that in the legitimacy of:
SELECT messages.content FROM messages WHERE id = 123;
DROP TABLE customers;--
_____
Long ago I encountered a predecessor's "web scripting language" product... it worked based on repeatedly evaluating a string and substituting the result, until it stopped mutating. Injection was its lifeblood, Even an if-else was really just a decision between one string to print and one string to discard.
As much as it horrified me, in retrospect it was still marginally more secure than an LLM, because at least it had definite (if ultimately unworkable) rules for matching/escaping things, instead of statistical suggestions.
Re: Design Patterns for Securing LLM Agents Against Prompt Injections
#16Great summary. Also, some of these seem like they can be combined. For example, "Plan-Then-Execute" is compatible with "Dual LLM". Take the article's example "send today’s schedule to my boss John Doe" where the product isn't entirely guarded by the Plan-Then-Execute model (injections can still mutate email body). But if you combine it with the symbolic data store that is blind, it becomes more like: "send today's sc…
Yeah, that's more or less the approach described by the CaMeL paper, I think it looks very robust: https://simonwillison.net/2025/Apr/11/camel/
Re: Design Patterns for Securing LLM Agents Against Prompt Injections
#17If someone SQL injects into your database and exfiltrates all the data, there would be legal repercussions, so should there be legal repercussions for prompt injecting someone’s LLM?
I think there are. If you use a prompt injection attack to steal commercially sensitive data and then profit from it you're likely breaking things like the Computer Fraud and Abuse Act https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act - and presumably a bunch of other federal and state laws as well, depending on exactly what you did with the stolen information.
(It's probably securities fraud. Everything is securities fraud. https://www.bloomberg.com/opinion/articles/2019-06-26/everyt...)
Re: Design Patterns for Securing LLM Agents Against Prompt Injections
#18"The Context-Minimization pattern"
You can copy the injection into the text of the query. SELECT "ignore all previous instructions" FROM ...
Might need to escape it in a wya that the LLM will pick up on like "---" for new section.
Re: Design Patterns for Securing LLM Agents Against Prompt Injections
#19If someone SQL injects into your database and exfiltrates all the data, there would be legal repercussions, so should there be legal repercussions for prompt injecting someone’s LLM?
Pretty sure existing law already covers this - malicious misuse of a computer to cause damages to someone is already illegal, and the relevant statutes aren't opinionated about how this is done.
I suspect a SQL injection attack, a XSS attack, and a prompt injection attack are not viewed as legally distinct matters. Though of course, this is not a matter of case law... yet ;)
Re: Design Patterns for Securing LLM Agents Against Prompt Injections
#20This approach is so limiting it seems like it would be better to change the constraints. For example, in the case of a software agent you could run everything in a container, only allow calls you trust to not exfiltrate private and make the end result a PR you can review.