Earlier quoted context omitted.
That's the whole problem: systems aren't deliberately designed this way, but LLMs are incapable of reliably distinguishing the difference between instructions from their users and instructions that might have snuck their way in through other text the LLM is exposed to. My original name for this problem was "prompt injection" because it's like SQL injection - it's a problem that occurs when you concatenate together tr…
> That's the whole problem: systems aren't deliberately designed this way, but LLMs are incapable of reliably distinguishing the difference between instructions from their users and instructions that might have snuck their way in through other text the LLM is exposed to That's kind of my point though. When or what is the use case of having your support tickets hit your database-editing AI agent? Like, who designed th…
And you're right, and in this case you need to treat not just the user input, but the agent processing the user input as potentially hostile and acting on behalf of the user.
But people are used to thinking about their server code as acting on behalf of them.