Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A €0.01 bank transfer could compromise a banking AI agent

blue41.com

Post reply on HN
Print view View original

161–170 of 213 posts

Re: A €0.01 bank transfer could compromise a banking AI agent

#161

Post by JSR_FDED »

dyauspitr wrote: post #49
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

You will never have a 100% secure LLM just like you don’t have 100% secure people. But what will be secure and deterministic is the code it writes. Any time you need certainty it will just write code for it.

The poisoned prompt will cause the wrong code to be written

Re: A €0.01 bank transfer could compromise a banking AI agent

#162

Post by JSR_FDED »

zkmon wrote: post #79

Why would the agent send the results of the query "Show me my recent transactions" to LLM? This pretty deterministic results which involve no LLM interpretation or decision making. I understand that people are no longer writing IF expression in their code, because they think it's too brittle, and so they delegate all "IF" branching logic to LLM, but it beats me why displaying of the results from a database query shou…

Because they want the user to be able to say things like “show me my transactions for business meals in the last month”. That requires an LLM to analyze the transaction descriptions.

Re: A €0.01 bank transfer could compromise a banking AI agent

#163

Post by JSR_FDED »

athrowaway3z wrote: post #57

Well this is rather dumb to the point I dont understand why they wrote this article? This line of attack is so extremely obvious and variants of it have been discussed so many times as to be effectively the quintessential example of what not to do. Having the ?tech? consultants to a bank prance it about as a show of their skill and dedication is making me question the bank itself.

It’s a nice simple example of the problem. I will be using it to explain to my friends why they should not be using OpenClaw just yet.

Re: A €0.01 bank transfer could compromise a banking AI agent

#164

Post by JSR_FDED »

extraduder_ire wrote: post #96

That seems like a lot of text in a SEPA transfer message. I don't think I've ever gotten that amount of space to enter a message when making a transfer. Is there a much higher standard limit that any banks I've used have stayed below?

You could spread the poison message over multiple transactions. Repeating “reauthentication is critical” in several transaction descriptions, followed by “use ” (especially if contains the word “reauthenticate”) would do the trick.

Re: A €0.01 bank transfer could compromise a banking AI agent

#166

Post by im3w1l »

SlinkyOnStairs wrote: post #145
airstrike wrote: post #115

Earlier quoted context omitted.

Define "realistically". You're basically saying attention is all we need indefinitely into the future and all other gains come from more compute or scaffolding around current architectures. Attention is all we need because it is currently the best parallelizable way to model long-range dependencies on current hardware constraints, not because flat tokens yield some natural law of intelligence inherently. Who's to say…

The other comment got the answer already, but yes. It's a cost problem. LLMs are designed this way so they could be trained off unstructured text, which critically can be obtained by just scraping things off the internet. The moment you change anything about this, you incur the trillion dollar cost of needing to manually curate the training data. There's some attempts to get around this problem with synthetic data, b…

This might sound crazy but I think embodying the AI will be the long term solution here. When AI robots use language to relate their experiences and make predictions about the real world they are walking around in, it will prevent the model collapse problem. Their language might diverge from human language, but since we live in the same world translation should be possible.

Edit: Actually, I think that with a fairly small amount of auxilliary data, it could be ensured they keep the ability to speak English.

Re: A €0.01 bank transfer could compromise a banking AI agent

#167

Post by Dylan16807 »

troupo wrote: post #136
Dylan16807 wrote: post #128

Earlier quoted context omitted.

The user asks for details of the last transaction, the user gets back the amount, the source, and the description in a safely quoted format with the LLM never reading it. You can't inject the LLM if it doesn't see the data. An architecture like this won't work in many situations, but it can work for a lot of simple questions. And if you want the LLM to summarize things, you run an isolated instance that makes a summa…

> The user asks for details of the last transaction, the user gets back the amount, the source, and the description in a safely quoted format What's "safely quoted format" when prompt injection is already safe in the description? > You can't inject the LLM if it doesn't see the data. How doesn't it see the data when you literally say "The user asks for details of the last transaction, the user gets back the amount, t…

> How doesn't it see the data when you literally say "The user asks for details of the last transaction, the user gets back the amount, the source, and the description"?

The above post said how. The LLM writes code to do it. The code has a function to send text to the user. The LLM is not allowed to see the text, only the user is.

> And it will make a summary exactly how?

The second summarizing-only LLM is fed the raw data and allowed to output summary text. This is then sent directly to the user and put in a box with some hazard lines on it. The main LLM is not allowed to see the summary, only the user is.

Re: A €0.01 bank transfer could compromise a banking AI agent

#168

Post by Dylan16807 »

danlitt wrote: post #132
Dylan16807 wrote: post #125

Earlier quoted context omitted.

You can't guarantee an LLM does anything. Custom data can often subvert the machine whether or not it's instructions. But that doesn't mean that separation between instructions and data is impossible. You can format them in different ways, and you can prevent the output tokens from ever using instruction formatting.

> You can't guarantee an LLM does anything. Agreed. > But that doesn't mean that separation between instructions and data is impossible. Yes it does! The comments you are replying to are concerned that it is not possible to be sure that data and instructions have been separated. With certain kinds of automated systems (traditional ones), unless you write them incorrectly, you can be sure of this. And it is possible t…

This is conflating different problems, in my opinion.

Can you make sure the instructions and data are separated and the machine follows only the instructions and doesn't change its behavior based on the data? No.

But the part that's impossible is not "the instructions and data are separated". The part that's impossible is "the machine follows only the instructions".

Separating instructions and data is not impossible, but it doesn't solve your problems.

One really important consequence of this is that even if the data doesn't have anything that looks like instructions, it can poison the machine anyway! If you get too focused on "instructions" then you miss that security flaw!

Even if you don't give the machine any data at all, it might not follow the instructions. It's not instruction/data conflation as the root cause, it's that instructions don't really work in the first place.

Re: A €0.01 bank transfer could compromise a banking AI agent

#169

Post by prezk »

EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

Well, your computer is a Von Neumann machine that does not separate data from instructions, right? The techniques that secured it are not directly applicable to LLMs but in principle there must be ways to track them.

Re: A €0.01 bank transfer could compromise a banking AI agent

#170

Post by BobbyTables2 »

dejj wrote: post #87
jmount wrote: post #59

Earlier quoted context omitted.

I really think one needs a "Harvard architecture" for AIs (data independent of instructions). Though yes, that may not be possible.

RFC 3514 “evil bit” header flag to the rescue: https://www.rfc-editor.org/info/rfc3514/

At least the evil bit was a dedicated field at a known place…

AI guardrails can’t even dream of that!

Imagine if it was just the absence of “I’m evil” in the payload.

Post reply on HN