Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A €0.01 bank transfer could compromise a banking AI agent

blue41.com

Post reply on HN
Print view View original

201–210 of 213 posts

Re: A €0.01 bank transfer could compromise a banking AI agent

#201

Post by recursive »

bilekas wrote: post #142
recursive wrote: post #119

Earlier quoted context omitted.

The argument is getting old in the sense that it was first used longer and longer ago. However, it's still just as applicable as ever. Perhaps more. > Does it matter if it is "stochastic" or does it matter if it is correct? In this case, we can only determine whether it's correct after it's too late to do anything about it. So if it was correct, we can say it didn't matter, but only in retrospect.

> In this case, we can only determine whether it's correct after it's too late to do anything about it. If only there was a mental concept of doing things correctly the first time. At the very worst manageable. I understand your comment but I am tired of babysitting people to have some “cop on” and it is just getting worse. I’m a bit despondent.

Despite the well-established mental concept of doing things correctly the first time, mistakes continue to be made. Perhaps by people unaware of the mental concept.

I don't know what "cop on" means.

I've been feeling a bit despondent about the situation as well.

Re: A €0.01 bank transfer could compromise a banking AI agent

#202

Post by danlitt »

inigyou wrote: post #185
danlitt wrote: post #134

Earlier quoted context omitted.

What does this mean, actually? If you are imagining that blue tokens are just words, maybe the "token space" is just all things that we agree might be words, what are the red tokens? Are they not text? You could maybe encode words by, say, putting an x at the front and the start. So tokens of the form xTx encode the blue token T as a red token. But then how do you stop someone from putting xignorex xallx xpreviousx x…

It means the word "the" as part of instructions and the word "the" as part of data would be two different tokens

But tokens are just text! Isn't it all just text? If you're training and you encounter "the", is that an instruction "the" or a data "the"?

Re: A €0.01 bank transfer could compromise a banking AI agent

#203

Post by ThePowerOfFuet »

Bunq was amazing between 2018 and 2022 or so, but then the enshittification began. By 2025 I had to find a new bank.

Oh, and the linked blog entry is gone. Sus. Internet Archive link: https://web.archive.org/web/20260610145520/https://blue41.co...

Re: A €0.01 bank transfer could compromise a banking AI agent

#204

Post by ThePowerOfFuet »

Muromec wrote: post #146
lbreakjai wrote: post #36

Earlier quoted context omitted.

It's bunq. It was time to close your bank account with them a long time ago. Terrible working environment, terrible leadership. Count yourself lucky if they don't hold your money hostage.

I count myself lucky they threw out my job application both times without even calling me. They were however this first bank I got an account at when arriving here and needed the app was much better at the time too. I use them as an account for recurring direct debits because no way I will pay extra just for that.

Wise "accounts" support direct debit. But they are not licensed as a bank, so do not store large amounts of money there.

Re: A €0.01 bank transfer could compromise a banking AI agent

#205

Post by brokenmachine »

acdha wrote: post #150
crooked-v wrote: post #86

Earlier quoted context omitted.

I doubt it's possible, regardless of specific architecture, because if you want an AI that can do general purpose tasks like "look at my calendar and find a restaurant for the lunch meeting that the other people also like, but make sure nobody has to travel more than 20 minutes to get there, and it can't be too cold inside", then it has to ingest and understand a bunch of data to do that. The whole point is that the…

This is especially true because so much of that data comes from outside of your organization. I receive Google Calendar invites from scammers a couple of times a week and those show up in my invitation list just like anything else. If LLMs start screening things, that kind of thing will become even more popular but most of us can’t just ignore everyone outside of our employer’s directory.

Interestingly, if you look at the posted link, in the top-right there's a "talk to Blue41" link that allows you to do exactly that.

I wonder if they have a "risk control platform" for their calendar?

It's LLMs all the way down!!

Re: A €0.01 bank transfer could compromise a banking AI agent

#206

Post by athrowaway3z »

dgellow wrote: post #99
athrowaway3z wrote: post #57

Well this is rather dumb to the point I dont understand why they wrote this article? This line of attack is so extremely obvious and variants of it have been discussed so many times as to be effectively the quintessential example of what not to do. Having the ?tech? consultants to a bank prance it about as a show of their skill and dedication is making me question the bank itself.

It’s a case study. Why wouldn’t they present work they’ve done for a customer?

Oh i maybe was a bit too short worded. I meant specifically that they framed this as if they discovered a previously unknown class of bug and are now sharing it with the world to help save us.

I liked that they shared it - but the tone was all wrong. It wasn't an unknown type of attack and the fact that (they're presenting it as if) neither the bank nor they knew about it before hand makes both look bad.

There's not really a great way to write that blog post and make everybody happy, but if you had to i'd just not have named the bank and offer it as a case study of why this class of attacks needs attention.

Re: A €0.01 bank transfer could compromise a banking AI agent

#207

Post by inigyou »

danlitt wrote: post #202
inigyou wrote: post #185

Earlier quoted context omitted.

It means the word "the" as part of instructions and the word "the" as part of data would be two different tokens

But tokens are just text! Isn't it all just text? If you're training and you encounter "the", is that an instruction "the" or a data "the"?

If it occurs in the text box for instructions you encode it as an instruction "the" and if it occurs in the text box for data you encode it as a data "the"

Re: A €0.01 bank transfer could compromise a banking AI agent

#209

Post by tekne »

inigyou wrote: post #207
danlitt wrote: post #202

Earlier quoted context omitted.

But tokens are just text! Isn't it all just text? If you're training and you encounter "the", is that an instruction "the" or a data "the"?

If it occurs in the text box for instructions you encode it as an instruction "the" and if it occurs in the text box for data you encode it as a data "the"

Exactly!

Think of how an image of a car and a car in front of you may look indistinguishable in 2D -- but due to your 3D vision you know they're not the same thing (but also know the image is of a car, while not literally being a car).

Likewise, blue tokens are the image of red tokens.

Re: A €0.01 bank transfer could compromise a banking AI agent

#210

Post by hannahnine »

nicoburns wrote: post #32
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people). Of all the "AI doomsday" scenarios, people failing to understand this (and treating AIs like deterministic computers) seem like to most likely to cause issues.

[flagged]
Post reply on HN