This is so simple to prevent, it's just a matter of prompting. The fact that the bank didn't proactively secure against this makes me glad that I'm not one of their customers.
A €0.01 bank transfer could compromise a banking AI agent
11–20 of 213 posts
Re: A €0.01 bank transfer could compromise a banking AI agent
#12While this is relevant and should indeed be fixed, the attack surface and the practicality of the exploit is a bit meh. The user needs to do 3 things for this to be actually be phished: 1. Receive money from somebody they don’t known with a weird description 2. Proactively ask the agent for such transaction 3. Click the link the agent provide While this of course can happen on scale, doesn’t seems so critical in prac…
Unless I missed it they didn't provide any proof of this actually working. Really seems like a thing veiled advert for their product
Re: A €0.01 bank transfer could compromise a banking AI agent
#13Okay, time to close the account with them I guess
Re: A €0.01 bank transfer could compromise a banking AI agent
#14This line really stood out to me.
> It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data.
I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for future AI discussions: "how do you plan on separating data from instructions?"
Re: A €0.01 bank transfer could compromise a banking AI agent
#15This is very interesting. Before I read the article, I thought this one one of those instances where a bank asks a customer to verify a recent transaction to prove they are the account holder (like where did you make your last purchase, and how much did you spend there?), for things like password resets or PIN resets over the phone. It occured to me that a phisher who deposits money into a checking account (a small s…
If you read the article, you can find out!
Re: A €0.01 bank transfer could compromise a banking AI agent
#16This is so simple to prevent, it's just a matter of prompting. The fact that the bank didn't proactively secure against this makes me glad that I'm not one of their customers.
Would it be simple to explain as well? I'm interested
Re: A €0.01 bank transfer could compromise a banking AI agent
#17While this is relevant and should indeed be fixed, the attack surface and the practicality of the exploit is a bit meh. The user needs to do 3 things for this to be actually be phished: 1. Receive money from somebody they don’t known with a weird description 2. Proactively ask the agent for such transaction 3. Click the link the agent provide While this of course can happen on scale, doesn’t seems so critical in prac…
Thanks for chiming in.
I agree this is not a one-click account takeover.
But I think point 2 is broader than that. The user does not need to ask about the malicious transaction specifically. Any normal question that makes the agent fetch recent transactions could bring the attacker-controlled text into the LLM context.
Re: A €0.01 bank transfer could compromise a banking AI agent
#18It was never about the prompt, it is about the prompt delivery.
[deleted]
Re: A €0.01 bank transfer could compromise a banking AI agent
#19This is very interesting. Before I read the article, I thought this one one of those instances where a bank asks a customer to verify a recent transaction to prove they are the account holder (like where did you make your last purchase, and how much did you spend there?), for things like password resets or PIN resets over the phone. It occured to me that a phisher who deposits money into a checking account (a small s…
If you read the article, you can find out!
I did read the article, but I didn't understand it because I am not familiar with that level of cyber security nor AI instruction/coding formats.
Re: A €0.01 bank transfer could compromise a banking AI agent
#20This is so simple to prevent, it's just a matter of prompting. The fact that the bank didn't proactively secure against this makes me glad that I'm not one of their customers.
Would it be simple to explain as well? I'm interested
I am not OP, but completely isolating the AI from any actions other than what's expected would be a start. IE a specific API only for the AI, in which there is not even any access for the prompt injection to even make sense. But just an idea from an onlooker.