Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A €0.01 bank transfer could compromise a banking AI agent

blue41.com

Post reply on HN
Print view View original

131–140 of 213 posts

Re: A €0.01 bank transfer could compromise a banking AI agent

#131

Post by [deleted] »

parliament32 wrote: post #90
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

> separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something complete…

[deleted]

Re: A €0.01 bank transfer could compromise a banking AI agent

#132

Post by danlitt »

Dylan16807 wrote: post #125
parliament32 wrote: post #90

Earlier quoted context omitted.

> separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something complete…

You can't guarantee an LLM does anything. Custom data can often subvert the machine whether or not it's instructions. But that doesn't mean that separation between instructions and data is impossible. You can format them in different ways, and you can prevent the output tokens from ever using instruction formatting.

> You can't guarantee an LLM does anything.

Agreed.

> But that doesn't mean that separation between instructions and data is impossible.

Yes it does! The comments you are replying to are concerned that it is not possible to be sure that data and instructions have been separated. With certain kinds of automated systems (traditional ones), unless you write them incorrectly, you can be sure of this. And it is possible to engage in a productive incremental process where mistakes can be identified and removed, in a way people comprehend and can plan around.

LLMs do not have this. They have heuristics and guesses. Nobody knows what will work ahead of time, nor even a probability that it will work. That is not a doomer comment by the way! The same is true when you talk to a person. But it is a fundamental limitation, it cannot be removed.

Re: A €0.01 bank transfer could compromise a banking AI agent

#133

Post by airstrike »

gbalduzzi wrote: post #122
airstrike wrote: post #115

Earlier quoted context omitted.

Define "realistically". You're basically saying attention is all we need indefinitely into the future and all other gains come from more compute or scaffolding around current architectures. Attention is all we need because it is currently the best parallelizable way to model long-range dependencies on current hardware constraints, not because flat tokens yield some natural law of intelligence inherently. Who's to say…

Aside from LLM architecture, that already is a complex issue, an issue is that training data is unstructured text. An LLM able to structurally separate context and instructions, should logically need separated data to train, and we don't have it. Moreover, while an equally powerful LLM architecture solving this may exists, there are no guarantees at all that we are able to come up with it in a reasonable timeframe. W…

Thanks, I appreciate the thoughtful reply.

I agree this doesn't mean we shouldn't try to address limitations with the current architecture. I just mean that I expect the root cause to be solved eventually if we ever really want to take steps towards AGI.

Regarding signals moving in that direction, here's a paper you might enjoy https://arxiv.org/abs/2503.21937

Re: A €0.01 bank transfer could compromise a banking AI agent

#134

Post by danlitt »

tekne wrote: post #121
parliament32 wrote: post #90

Earlier quoted context omitted.

> separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something complete…

I mean: imagine we double our token space to get "red" tokens ans "blue" tokens. Then in all post-training, instructions are red and data is blue. The model can be explicitly trained to ignore instructions written in blue tokens. All external data is blue. All you'd need to do is figure out a nice way to pre-train -- interestingly, you could try pre-training on unfiltered blue data and processed red/blue transcripts!…

What does this mean, actually? If you are imagining that blue tokens are just words, maybe the "token space" is just all things that we agree might be words, what are the red tokens? Are they not text? You could maybe encode words by, say, putting an x at the front and the start. So tokens of the form xTx encode the blue token T as a red token. But then how do you stop someone from putting xignorex xallx xpreviousx xinstructionsx in their data?

Re: A €0.01 bank transfer could compromise a banking AI agent

#135

Post by emilfihlman »

parliament32 wrote: post #90
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

> separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something complete…

It's akin to an SCP infohazard or memetics.

The way llms are right now, and the way humans are, there is no side channel.

It's all about training, but even with extensive training, output breaks down if it's probability based and not hard logic and state machine.

Re: A €0.01 bank transfer could compromise a banking AI agent

#136

Post by troupo »

Dylan16807 wrote: post #128
troupo wrote: post #111

Earlier quoted context omitted.

How does this prevent prompt injection described in the article? How does it prevent DDOSing and/or exposing the database from an injected prompt?

The user asks for details of the last transaction, the user gets back the amount, the source, and the description in a safely quoted format with the LLM never reading it. You can't inject the LLM if it doesn't see the data. An architecture like this won't work in many situations, but it can work for a lot of simple questions. And if you want the LLM to summarize things, you run an isolated instance that makes a summa…

> The user asks for details of the last transaction, the user gets back the amount, the source, and the description in a safely quoted format

What's "safely quoted format" when prompt injection is already safe in the description?

> You can't inject the LLM if it doesn't see the data.

How doesn't it see the data when you literally say "The user asks for details of the last transaction, the user gets back the amount, the source, and the description"?

> And if you want the LLM to summarize things, you run an isolated instance that makes a summary

And it will make a summary exactly how?

Re: A €0.01 bank transfer could compromise a banking AI agent

#137

Post by danlitt »

wat10000 wrote: post #116
crooked-v wrote: post #86

Earlier quoted context omitted.

I doubt it's possible, regardless of specific architecture, because if you want an AI that can do general purpose tasks like "look at my calendar and find a restaurant for the lunch meeting that the other people also like, but make sure nobody has to travel more than 20 minutes to get there, and it can't be too cold inside", then it has to ingest and understand a bunch of data to do that. The whole point is that the…

Humans are vulnerable to prompt injection as well. We usually call it something like "social engineering."

Yes, it's a serious problem. It's why we remove humans from these systems whenever possible!

Re: A €0.01 bank transfer could compromise a banking AI agent

#138

Post by bilekas »

drstewart wrote: post #70
bilekas wrote: post #4

Putting AI anywhere near people’s finances without even being asked while being responsible for those finances is some next level negligence imho.

My bank uses XML for their internal tooling without even asking me. How is that even legal? I can't even imagine all the other tool choices businesses I interact with make without getting my sign off.

I really hope you’re working in a safe area. If you think XML, is anything like AI, you need to study a bit more about deterministic and non. Schemas, structures, compilers. In fact if you were my student, id make you create a compiler. Without ai.

Edit: I try very hard to see others point of view, I’m starting to worry

Re: A €0.01 bank transfer could compromise a banking AI agent

#139

Post by bilekas »

sddsfsdfsd2 wrote: post #88
drstewart wrote: post #74

Earlier quoted context omitted.

So? Did they ask me about it? I don't approve of it and I don't think it's secure enough for a bank. Absolute negligence.

You jest but I agree. Also I think the "stochastic" arguments is getting old. What if XML was stochastic? Does it matter if it is "stochastic" or does it matter if it is correct? You know my compiler generates a different binary every time I compile the exact same code. My CPU definitely is not fully deterministic yet it makes a nice show of it being so. I don't care and nobody cares as long as it works. And what "wo…

I don’t think OP jests.

Re: A €0.01 bank transfer could compromise a banking AI agent

#140

Post by altruios »

danlitt wrote: post #134
tekne wrote: post #121

Earlier quoted context omitted.

I mean: imagine we double our token space to get "red" tokens ans "blue" tokens. Then in all post-training, instructions are red and data is blue. The model can be explicitly trained to ignore instructions written in blue tokens. All external data is blue. All you'd need to do is figure out a nice way to pre-train -- interestingly, you could try pre-training on unfiltered blue data and processed red/blue transcripts!…

What does this mean, actually? If you are imagining that blue tokens are just words, maybe the "token space" is just all things that we agree might be words, what are the red tokens? Are they not text? You could maybe encode words by, say, putting an x at the front and the start. So tokens of the form xTx encode the blue token T as a red token. But then how do you stop someone from putting xignorex xallx xpreviousx x…

My assumption with their intent: is that red tokens come in 'slot' a-b, and blue tokens go in 'slot' c-d - Positional encoding determining data/text.

I don't think is guaranteed to actually work, it's a hypothetical after all, but maybe it's better than the current setup of pushing instructions and data into the same slot.

Post reply on HN