Well this is rather dumb to the point I dont understand why they wrote this article? This line of attack is so extremely obvious and variants of it have been discussed so many times as to be effectively the quintessential example of what not to do. Having the ?tech? consultants to a bank prance it about as a show of their skill and dedication is making me question the bank itself.
A €0.01 bank transfer could compromise a banking AI agent
101–110 of 213 posts
Re: A €0.01 bank transfer could compromise a banking AI agent
#102Could we fix the title to match the article? > How we helped Bunq secure their financial AI assistant
I think the current title, while admittedly a bit clickbaity, describes the core issue better.
Fair enough, my point is mostly that it doesn’t follow the HN guidelines:
> Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.
The current one is editorialized and clickbait-ish
Re: A €0.01 bank transfer could compromise a banking AI agent
#103Earlier quoted context omitted.
I really think one needs a "Harvard architecture" for AIs (data independent of instructions). Though yes, that may not be possible.
It's not possible with today's LLM models, but we are not wedded to the current architecture.
Realistically, we are.
This is not some arbitrary design choice, it's the core compromise to make LLMs viable to train at all.
Re: A €0.01 bank transfer could compromise a banking AI agent
#104The name of the agent is 'finn' - is that a reference to Intercom's Fin agent?
Re: A €0.01 bank transfer could compromise a banking AI agent
#105This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…
It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people). Of all the "AI doomsday" scenarios, people failing to understand this (and treating AIs like deterministic computers) seem like to most likely to cause issues.
I can't believe that fucking Terminator was prophetic.
Re: A €0.01 bank transfer could compromise a banking AI agent
#106Re: A €0.01 bank transfer could compromise a banking AI agent
#107Earlier quoted context omitted.
It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people). Of all the "AI doomsday" scenarios, people failing to understand this (and treating AIs like deterministic computers) seem like to most likely to cause issues.
> It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Seems solved already? Exactly what the system/user division is about, and if that's not enough for you, use a model that has a developer/system/user divide. Today's SOTA LLMs have pretty excellent following of these divisions, and the user "instructions", regardless if they're smuggled in, won't override the system ones. The difficul…
> whitelist what's possible
Why do you need LLMs in the first place if you are whitelisting possible inputs?
You can use a much simpler and less costly system.
Re: A €0.01 bank transfer could compromise a banking AI agent
#108Earlier quoted context omitted.
So? Did they ask me about it? I don't approve of it and I don't think it's secure enough for a bank. Absolute negligence.
You jest but I agree. Also I think the "stochastic" arguments is getting old. What if XML was stochastic? Does it matter if it is "stochastic" or does it matter if it is correct? You know my compiler generates a different binary every time I compile the exact same code. My CPU definitely is not fully deterministic yet it makes a nice show of it being so. I don't care and nobody cares as long as it works. And what "wo…
I don't know what compiler you're using but our C/C++ builds are definitely deterministic given the same source files. We have CI tests to ensure this remains the case.
Re: A €0.01 bank transfer could compromise a banking AI agent
#109Earlier quoted context omitted.
> It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Seems solved already? Exactly what the system/user division is about, and if that's not enough for you, use a model that has a developer/system/user divide. Today's SOTA LLMs have pretty excellent following of these divisions, and the user "instructions", regardless if they're smuggled in, won't override the system ones. The difficul…
> whitelist what's possible Why do you need LLMs in the first place if you are whitelisting possible inputs? You can use a much simpler and less costly system.
There is like a billion use cases out there, lord knows why some people do some stuff. There are more use cases than just "creative text" or free-form outputs, lots of other things, paired together with an harness too. Like an support agent even perhaps.
Re: A €0.01 bank transfer could compromise a banking AI agent
#110> Modern banking apps increasingly include AI-powered features. These sit between the user and a range of backend data sources, such as transaction records, product documentation, account details
Literally no one stopped to even question the insanity of this. "just add more AI"