Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A €0.01 bank transfer could compromise a banking AI agent

blue41.com

Post reply on HN
Print view View original

121–130 of 213 posts

Re: A €0.01 bank transfer could compromise a banking AI agent

#121

Post by tekne »

parliament32 wrote: post #90
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

> separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something complete…

I mean: imagine we double our token space to get "red" tokens ans "blue" tokens.

Then in all post-training, instructions are red and data is blue. The model can be explicitly trained to ignore instructions written in blue tokens. All external data is blue.

All you'd need to do is figure out a nice way to pre-train -- interestingly, you could try pre-training on unfiltered blue data and processed red/blue transcripts!

Likewise, model-actions (e.g. open file) could be written only in red, and hence you'd never learn to do them from the unfiltered data.

The only connection between the red world and the blue world would be the processed trainign chats containing red and blue data togethers -- allowing the model to learn the relationship between them (while only being exposed to examples where red instructions are strictly followed, whatever the blue says)

Re: A €0.01 bank transfer could compromise a banking AI agent

#122

Post by gbalduzzi »

airstrike wrote: post #115
SlinkyOnStairs wrote: post #103

Earlier quoted context omitted.

Realistically, we are. This is not some arbitrary design choice, it's the core compromise to make LLMs viable to train at all .

Define "realistically". You're basically saying attention is all we need indefinitely into the future and all other gains come from more compute or scaffolding around current architectures. Attention is all we need because it is currently the best parallelizable way to model long-range dependencies on current hardware constraints, not because flat tokens yield some natural law of intelligence inherently. Who's to say…

Aside from LLM architecture, that already is a complex issue, an issue is that training data is unstructured text.

An LLM able to structurally separate context and instructions, should logically need separated data to train, and we don't have it.

Moreover, while an equally powerful LLM architecture solving this may exists, there are no guarantees at all that we are able to come up with it in a reasonable timeframe.

Without some signals moving in that direction, the most pragmatic and realistic way of looking at the problem is that it will not be solved in the near future

Re: A €0.01 bank transfer could compromise a banking AI agent

#123

Post by ozim »

EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

Quite simple you make harness and loads of people are building harnesses as we speak.

Right now also a lot of people are building in a way where they give a sample data to LLM so that AI agent builds deterministic code for crunching data so that actual data doesn't go to LLM and is processd by regular code, only that code for processing is written by agent.

You can always process only descriptions that are in the list and ones that are not recognized "ask a human" so just an allowlist. I do believe normal person would have most transactions that would be mostly the same and then couple that would stand out so you also can make allowlist from last 2 years as a starting point, not to bother people too much (I think no one has prompt injection in their last 2 years banking history besides ultra nerds maybe).

I think by now it is common knowledge that "just dump all data at LLM and as some questions" or "let LLM process anything someone sends me in an e-mail" is silly.

In "the standoff" Pliny was trying to hack tszzl harness and it wasn't working an Pliny is notorious for jail breaking LLMs.

Re: A €0.01 bank transfer could compromise a banking AI agent

#124

Post by AceJohnny2 »

EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

> "how do you plan on separating data from instructions?"

Use a Harvard Architecture CPU, duh

https://en.wikipedia.org/wiki/Harvard_architecture

(j/k, if it wasn't obvious)

Re: A €0.01 bank transfer could compromise a banking AI agent

#125

Post by Dylan16807 »

parliament32 wrote: post #90
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

> separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something complete…

You can't guarantee an LLM does anything. Custom data can often subvert the machine whether or not it's instructions.

But that doesn't mean that separation between instructions and data is impossible. You can format them in different ways, and you can prevent the output tokens from ever using instruction formatting.

Re: A €0.01 bank transfer could compromise a banking AI agent

#126

Post by totetsu »

ozim wrote: post #123
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

Quite simple you make harness and loads of people are building harnesses as we speak. Right now also a lot of people are building in a way where they give a sample data to LLM so that AI agent builds deterministic code for crunching data so that actual data doesn't go to LLM and is processd by regular code, only that code for processing is written by agent. You can always process only descriptions that are in the lis…

I’ve noticed that for task that require consistency across very large body of text, like translating strings of very large doc, the approach of letting the agent split and it up and programmatically do it bit by bit, is much worse quality than just dumping it all in a single llm context.

Re: A €0.01 bank transfer could compromise a banking AI agent

#127

Post by ianburrell »

nerder92 wrote: post #6

While this is relevant and should indeed be fixed, the attack surface and the practicality of the exploit is a bit meh. The user needs to do 3 things for this to be actually be phished: 1. Receive money from somebody they don’t known with a weird description 2. Proactively ask the agent for such transaction 3. Click the link the agent provide While this of course can happen on scale, doesn’t seems so critical in prac…

This is similar to scam where people are sent messages about bad transaction with a fake link to the bank to verify it. Some attackers have gotten Paypal to send notifications that have the link. People are supposed to check the source and go directly to bank, and this will bypass that.

Re: A €0.01 bank transfer could compromise a banking AI agent

#128

Post by Dylan16807 »

troupo wrote: post #111
dyauspitr wrote: post #55

Earlier quoted context omitted.

[flagged]

How does this prevent prompt injection described in the article? How does it prevent DDOSing and/or exposing the database from an injected prompt?

The user asks for details of the last transaction, the user gets back the amount, the source, and the description in a safely quoted format with the LLM never reading it.

You can't inject the LLM if it doesn't see the data.

An architecture like this won't work in many situations, but it can work for a lot of simple questions.

And if you want the LLM to summarize things, you run an isolated instance that makes a summary and you never show that summary to the LLM that's following the user's instructions.

Re: A €0.01 bank transfer could compromise a banking AI agent

#129

Post by ozim »

totetsu wrote: post #126
ozim wrote: post #123

Earlier quoted context omitted.

Quite simple you make harness and loads of people are building harnesses as we speak. Right now also a lot of people are building in a way where they give a sample data to LLM so that AI agent builds deterministic code for crunching data so that actual data doesn't go to LLM and is processd by regular code, only that code for processing is written by agent. You can always process only descriptions that are in the lis…

I’ve noticed that for task that require consistency across very large body of text, like translating strings of very large doc, the approach of letting the agent split and it up and programmatically do it bit by bit, is much worse quality than just dumping it all in a single llm context.

I guess someone is doing harness for that use case then. I was mostly thinking about payment transfer description that mostly would be more like a sentence. More about data lines like CSV as that would be what is used in banking.

Lots of known attacks can be found with static analysis of text, even in long text blocks, finding "unexpected characters", finding "white text on white background" will still prevent a lot of attacks I believe. If you find in a text any IOC just don't process the text, write it to log file, document and let some person make a decision.

Re: A €0.01 bank transfer could compromise a banking AI agent

#130

Post by _pdp_ »

I can only speculate why this is possible but if I had to guess it is due to the fact that the external messages are effectively added as "user" type thus appear as direct instructions.

And this is far much common then one might think and classic problem across the board. There are easy solutions too.

Post reply on HN