Earlier quoted context omitted.
Yes, I know it was an example, I was just running with it because it's a convenient example. My point is that we've known for a couple decades at least that letting user input touch your production, unfiltered and unsanitized, is bad. The same concept as SQL exists with user-generated AI input. Sanitize input, map input to known/approved outputs, robust security boundaries, etc. Yet, for some reason, every week there…
But how can you sanitize text? That's what makes this stuff hard: the previous lessons we have learned about web application security don't entirely match up to how LLMs work. If you show me an app with a SQL injection hole or XSS hole, I know how to fix it. If your app has a prompt injection hole, the answer may turn out to be "your app is fundamentally insecure and cannot be built safely". Nobody wants to hear that…
But, in the CaMel proposal example, what prevents malicious instructions in the un-trusted content returning an email address that is in the trusted contacts list, but is not the correct one?
This situation is less concerning, yes, but generally, how would you prevent instructions that attempt to reduce the accuracy of parsing, for example, while not actually doing anything catastrophic