Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A €0.01 bank transfer could compromise a banking AI agent

blue41.com

Post reply on HN
Print view View original

31–40 of 213 posts

Re: A €0.01 bank transfer could compromise a banking AI agent

#31

Post by tvissers »

bilekas wrote: post #20
jorisw wrote: post #16

Earlier quoted context omitted.

Would it be simple to explain as well? I'm interested

I am not OP, but completely isolating the AI from any actions other than what's expected would be a start. IE a specific API only for the AI, in which there is not even any access for the prompt injection to even make sense. But just an idea from an onlooker.

I can recommend having a look at secure design patterns for LLM agents. Simon Willison has a great post on this: https://simonwillison.net/2025/Jun/13/prompt-injection-desig...

Re: A €0.01 bank transfer could compromise a banking AI agent

#32

Post by nicoburns »

EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people).

Of all the "AI doomsday" scenarios, people failing to understand this (and treating AIs like deterministic computers) seem like to most likely to cause issues.

Re: A €0.01 bank transfer could compromise a banking AI agent

#34

Post by crote »

nemomarx wrote: post #22
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

Is there any good tech for it, though? This just seems like an inherent language model behavior and at best everyone has guard rails or big exclamation marks to separate their own instructions a little.

Correct. It should've been an immediate dealbreaker for applying the current generation of LLMs in crucial environments like banking.

Unfortunately we live in a world where the CxO cares more about playing "keeping up with the Joneses" with his golf buddies and seeing the share price do a little bump every time he mentions AI. Truly keeping your money secure is not even remotely a priority.

Re: A €0.01 bank transfer could compromise a banking AI agent

#35

Post by repelsteeltje »

addandsubtract wrote: post #25
bilekas wrote: post #20

Earlier quoted context omitted.

I am not OP, but completely isolating the AI from any actions other than what's expected would be a start. IE a specific API only for the AI, in which there is not even any access for the prompt injection to even make sense. But just an idea from an onlooker.

Now that you mention it, why don't we encrypt injectable data that comes from users and only decrypt it on the client?

You mean, use encryption (+base64 or something) as a "poor man's" string-escape? Interesting idea!

Re: A €0.01 bank transfer could compromise a banking AI agent

#36

Post by lbreakjai »

Muromec wrote: post #13

Okay, time to close the account with them I guess

It's bunq. It was time to close your bank account with them a long time ago. Terrible working environment, terrible leadership.

Count yourself lucky if they don't hold your money hostage.

Re: A €0.01 bank transfer could compromise a banking AI agent

#37

Post by Angostura »

nicoburns wrote: post #32
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people). Of all the "AI doomsday" scenarios, people failing to understand this (and treating AIs like deterministic computers) seem like to most likely to cause issues.

Jokes on them. My bank will just truncate it to 10 characters.

Re: A €0.01 bank transfer could compromise a banking AI agent

#38

Post by crote »

cowlby wrote: post #30

Defense in depth approach, would this work to help as a layer? - Wrap user input in strong markers like - Have the agent compute what it will perform as structured output. - Have another agent evaluate the structured output against the intent of the code. - Determine if it aligns or deviates from the intended workflow. Execute or deny gate from here.

No, you're still just one clever prompt away from getting pwned. It's like trying to solve SQL injection by attempting to use an ever-increasing pile of regexes for "input validation", rather than just getting rid of string concatenation and using prepared statements instead.

Re: A €0.01 bank transfer could compromise a banking AI agent

#39

Post by NitpickLawyer »

reddalo wrote: post #3

Good job AI, after we managed to almost fix SQL injections everywhere, you made them come back!

That's precisely why I am using a different analogy when talking about this. The SQL injection analogy only matches the injection part, not the rest. There is nothing to secure, because there is no SQL query. You want the agent to work on data, in a "general" way, otherwise you'd just use a script.

The better analogy is phishing. Because that's what's happening here. The "prompt injection" attack is trying to "phish" the LLM into doing something unintended. That's how we should all comunicate it, as it matches better with what's happening. Unfortunately there aren't really good defences for it, as we all know from phishing "education" / "campaigns". Your best bet is to secure it in layers, try to have warnings (i.e. classification models) you try to secure the next step (i.e. capabilities based tool execution) and so on. But it's not foolproof and it should be communicated clearly.

Re: A €0.01 bank transfer could compromise a banking AI agent

#40

Post by csomar »

nerder92 wrote: post #6

While this is relevant and should indeed be fixed, the attack surface and the practicality of the exploit is a bit meh. The user needs to do 3 things for this to be actually be phished: 1. Receive money from somebody they don’t known with a weird description 2. Proactively ask the agent for such transaction 3. Click the link the agent provide While this of course can happen on scale, doesn’t seems so critical in prac…

People already click suspicious emails that ask them to login. At a high number of attempts, some chickens will be caught. However, people are now weary of emails since there is a lot of phishing there. On the other hand, the AI assistant env. could be considered "safe" by users because it's stuff coming from the bank. So they are more likely to fall for it. (honestly, unless you are a dev and aware of prompt injection, I don't see why the users wouldn't fall for it).
Post reply on HN