Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Design Patterns for Securing LLM Agents Against Prompt Injections

simonwillison.net

Post reply on HN
Print view View original

1–10 of 31 posts

Re: Design Patterns for Securing LLM Agents Against Prompt Injections

#4

Post by swyx »

ooh this is a dense and useful paper. i like that they took the time to apply it to a bunch of case studies and its all in 30 pages.

i think basically all of them involve reducing the "agency" of the agents though - which is a fine tradeoff - but i think one should be aware that the Big Model folks dont try to engineer any of these and just collect data to keep reducing injection risk. the tradeoff of capability maxxing vs efficiency/security often tends to be won by the capabilitymaxxers in terms of product adoption/marketing.

eg the SWE Agent case study recommends Dual LLM with strict data formatting - would like to see this benchmarked in terms of how much of a perfomance an agent like this would be, perhaps doable by forking openai codex and implementing the dual llm.

Re: Design Patterns for Securing LLM Agents Against Prompt Injections

#5

Post by simonw »

swyx wrote: post #4

ooh this is a dense and useful paper. i like that they took the time to apply it to a bunch of case studies and its all in 30 pages. i think basically all of them involve reducing the "agency" of the agents though - which is a fine tradeoff - but i think one should be aware that the Big Model folks dont try to engineer any of these and just collect data to keep reducing injection risk. the tradeoff of capability maxx…

Yeah, this paper is refreshingly conservative and practical: it takes the position that robust protection against prompt injection requires very painful trade-offs:

  These patterns impose intentional
  constraints on agents, explicitly 
  limiting their ability to perform 
  arbitrary tasks.
That's a bucket of cold water in a lot of things people are trying to build. I imagine a lot of people will ignore this advice!

Re: Design Patterns for Securing LLM Agents Against Prompt Injections

#6

Post by simonw »

My favorite line from this paper:

> The design patterns we propose share a common guiding principle: once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions—that is, actions with negative side effects on the system or its environment.

This is the key thing people need to understand about why prompt injection is such a critical issue, especially now everyone is wiring LLMs together with tools and MCP servers and building "agents".

Re: Design Patterns for Securing LLM Agents Against Prompt Injections

#7

Post by NoMoreNicksLeft »

simonw wrote: post #5
swyx wrote: post #4

ooh this is a dense and useful paper. i like that they took the time to apply it to a bunch of case studies and its all in 30 pages. i think basically all of them involve reducing the "agency" of the agents though - which is a fine tradeoff - but i think one should be aware that the Big Model folks dont try to engineer any of these and just collect data to keep reducing injection risk. the tradeoff of capability maxx…

Yeah, this paper is refreshingly conservative and practical: it takes the position that robust protection against prompt injection requires very painful trade-offs: These patterns impose intentional constraints on agents, explicitly limiting their ability to perform arbitrary tasks. That's a bucket of cold water in a lot of things people are trying to build. I imagine a lot of people will ignore this advice!

LLMs are too useful to allow the commoner access to them. The question remains, how best to fleece those commoners with perceived utility while providing them none?

Re: Design Patterns for Securing LLM Agents Against Prompt Injections

#9

Post by simonw »

JSR_FDED wrote: post #3

Clever. It’s like parameterized queries for SQL.

If only it were as easy as that!

The problem with prompt injection is that the attack itself is the same as SQL injection - concatenation trusted and untrusted strings together - but so far all of our attempts at implementing a solution similar to parameterized queries (such as system prompts and prompt delimiters) have failed.

Re: Design Patterns for Securing LLM Agents Against Prompt Injections

#10

Post by hooverd »

swyx wrote: post #4

ooh this is a dense and useful paper. i like that they took the time to apply it to a bunch of case studies and its all in 30 pages. i think basically all of them involve reducing the "agency" of the agents though - which is a fine tradeoff - but i think one should be aware that the Big Model folks dont try to engineer any of these and just collect data to keep reducing injection risk. the tradeoff of capability maxx…

What if we could define what a computer could do via some symbolic notation? Perhaps program it in some kind of language?
Post reply on HN