111–120 of 213 posts
Why would the agent send the results of the query "Show me my recent transactions" to LLM? This pretty deterministic results which involve no LLM interpretation or decision making. I understand that people are no longer writing IF expression in their code, because they think it's too brittle, and so they delegate all "IF" branching logic to LLM, but it beats me why displaying of the results from a database query shou…
You'd of course need another one to check the sanity of the sanity check decision of the previous one.
This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…
The idea is not to distinguish instructions from data. It is closer to having different privilege levels. Not all code has to run in kernel space, some code runs in unprivileged user space. So what is the equivalent for LLM agents?
In FIDES-style systems, every piece of information that enters the agent context is labeled along two dimensions: integrity and confidentiality. Integrity captures whether the data is trusted or untrusted (i.e. could it contain a prompt injection attack). Confidentiality captures who is allowed to see or receive it [0].
The privileged agent, sometimes called the planning agent, should not directly see untrusted data because it would be susceptible to prompt injection attacks. In the article’s example, a bank transaction’s sender-supplied reference would be untrusted. Instead, the planning agent receives a variable token. It can then either delegate processing of that variable to an unprivileged / quarantined agent with no or limited tool access, or pass the token as a reference to a tool.
Tools then have policies attached to their arguments and outputs. These policies specify which integrity and confidentiality levels are allowed, and whether the tool call may proceed. The policy also determines how the result should be labeled.
For example:
1. High-confidentiality data should not be allowed to flow into a `send_email` tool call addressed to an external recipient.
2. A tool call whose result depends on untrusted input should generally produce untrusted output.
3. A sensitive side-effecting tool should be able to reject calls that are influenced by untrusted context.
So the answer to “how do you separate data from instructions?” may be: you do not rely on the model to do that separation. You track provenance and privilege outside the model, and then enforce the security policy at the tool boundary.
[0] In the simplest implementation, confidentiality is assessed with a binary low/high value, however, in a more advanced implementation, confidentiality can be represented as the set of users or principals allowed to learn that information.
Earlier quoted context omitted.
It's not possible with today's LLM models, but we are not wedded to the current architecture.
Realistically, we are. This is not some arbitrary design choice, it's the core compromise to make LLMs viable to train at all .
Attention is all we need because it is currently the best parallelizable way to model long-range dependencies on current hardware constraints, not because flat tokens yield some natural law of intelligence inherently.
Who's to say we won't find a way to encode provenance or privilege natively into models such that the tradeoff changes?
It's hard to say what the solution will be. If I knew it, I'd build it. But it's even harder to sustain that the current architecture is a crystalized global optimum.
Earlier quoted context omitted.
I really think one needs a "Harvard architecture" for AIs (data independent of instructions). Though yes, that may not be possible.
I doubt it's possible, regardless of specific architecture, because if you want an AI that can do general purpose tasks like "look at my calendar and find a restaurant for the lunch meeting that the other people also like, but make sure nobody has to travel more than 20 minutes to get there, and it can't be too cold inside", then it has to ingest and understand a bunch of data to do that. The whole point is that the…
This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…
> separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something complete…
It can be done, but unsurprisingly it looks exactly like microservices distributed auth (also ZTP).
It's all the same problem, just instead of a JVM, it's an LLM.
Earlier quoted context omitted.
> separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something complete…
Right, you have to set boundaries. You put each task and user input into a box, and then the LLM makes a decision. It can only access APIs that have user identity attached, that act within the scope of the requesting user. It can be done, but unsurprisingly it looks exactly like microservices distributed auth (also ZTP). It's all the same problem, just instead of a JVM, it's an LLM.
Like in the banking world, you can make everything super authenticated, but if you have an API that receives the latest wire transfer YOU received with the message attached, you don't control the message content and it can be an attack vector.
Being authenticated/authorized is not the solution, it is data that the user can access.
Earlier quoted context omitted.
So? Did they ask me about it? I don't approve of it and I don't think it's secure enough for a bank. Absolute negligence.
You jest but I agree. Also I think the "stochastic" arguments is getting old. What if XML was stochastic? Does it matter if it is "stochastic" or does it matter if it is correct? You know my compiler generates a different binary every time I compile the exact same code. My CPU definitely is not fully deterministic yet it makes a nice show of it being so. I don't care and nobody cares as long as it works. And what "wo…
However, it's still just as applicable as ever. Perhaps more.
> Does it matter if it is "stochastic" or does it matter if it is correct?
In this case, we can only determine whether it's correct after it's too late to do anything about it. So if it was correct, we can say it didn't matter, but only in retrospect.
Earlier quoted context omitted.
That's precisely why I am using a different analogy when talking about this. The SQL injection analogy only matches the injection part, not the rest. There is nothing to secure, because there is no SQL query. You want the agent to work on data, in a "general" way, otherwise you'd just use a script. The better analogy is phishing. Because that's what's happening here. The "prompt injection" attack is trying to "phish"…
> There is nothing to secure, because there is no SQL query. Yet.