91–100 of 213 posts
Earlier quoted context omitted.
Jokes on them. My bank will just truncate it to 10 characters.
> Jokes on them. My bank will just truncate it to 10 characters. You do understand that this is just an example out of a bazillion and that planning to solve every place where data is fed to LLMs at 10 characters so that it's not mistaken for instructions ain't a viable solution?
Earlier quoted context omitted.
It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Ultimately the only protection is to limit the powers we grant to any given LLM to reduce the fallout when (not if) things go wrong (much like we do with people). Of all the "AI doomsday" scenarios, people failing to understand this (and treating AIs like deterministic computers) seem like to most likely to cause issues.
I really think one needs a "Harvard architecture" for AIs (data independent of instructions). Though yes, that may not be possible.
Earlier quoted context omitted.
I've seen it quite a lot in my career: even when prepared statements are available and easy to use from a SQL client library, many programmers will simply not use them, in favor of format strings and string concatenation (maybe with an attempt to quote/escape user input). Just having support for the right way isn't enough. You have to put up roadblocks when people try to go the wrong way.
Why is a format string or string concatenation (or interpolation, what I would use) the “wrong way” when all user input (more precisely: all string literals) are properly escaped?
There's an incidental performance benefit on some database engines as well. When you write a SQL query, in general the database engine has to compile this to a form it can use
If you use raw string concatenation, "SELECT USERS FROM table WHERE id=1" might compile to something like (pseudocode below)
def prepstatement1():
...
def prepstatement2(id: int): # [1]: For example SQL Server has xp_cmdshell: https://learn.microsoft.com/en-us/sql/relational-databases/s...
Earlier quoted context omitted.
Why not write some wrapper code so you can basically hand the LLM placeholders for data it never gets to see? Whenever it uses the placeholder in the response, you replace it with the real data (via real code, not by telling an LLM to "do that"). Surely this has been tried? If so, what makes it not work, or work badly? I'm honestly curious.
Fundamentally, an LLM is a list of N tokens that generates N+1 tokens. In other words, it's just a wall of text (aka context window). There's no way to tell it "tokens 124 through 200 are dangerous, please disregard those" except by putting words into the context window. So the placeholders and the instructions both coexist in the context window, and one can override the other. In other words, if you have placeholder…
Hence "real code"
You have some markup for secret start/end. Instead of passing the input directly to the LLM, you parse it first, take anything within "secret/dangerous tags" and store it, generate a key for it and put that key where the secret was, then you pass it on to the LLM. Let's say the work of the LLM is "give me (not "make") the POST request to make the bank transaction", you get a response, replace the keys with the secrets in the response, and make the POST request.
I'm sure there's a million interesting ways this could fail or be useless [0], but passing user input or a secret to the LLM would never, ever happen.
[0] if LLM suck at math, they may suck at reproducing lots of long hashes 100% correctly, too? I have no idea
Earlier quoted context omitted.
> It seems to me like it's a fundamentally unsolvable architectural issue with LLMs. Seems solved already? Exactly what the system/user division is about, and if that's not enough for you, use a model that has a developer/system/user divide. Today's SOTA LLMs have pretty excellent following of these divisions, and the user "instructions", regardless if they're smuggled in, won't override the system ones. The difficul…
If it was solved, the bug like this would not happen. It is also not always clear who is the user and how much they should be obeyed
Only if you only read the first line in my comment, there is more under that one too.
It is clear, if you make it clear. These bugs happen because they don't clearly understand what should go where.
Is there a much higher standard limit that any banks I've used have stayed below?
> How we helped Bunq secure their financial AI assistant
Earlier quoted context omitted.
Fundamentally, an LLM is a list of N tokens that generates N+1 tokens. In other words, it's just a wall of text (aka context window). There's no way to tell it "tokens 124 through 200 are dangerous, please disregard those" except by putting words into the context window. So the placeholders and the instructions both coexist in the context window, and one can override the other. In other words, if you have placeholder…
> There's no way to tell it "tokens 124 through 200 are dangerous, please disregard those" Hence "real code" You have some markup for secret start/end. Instead of passing the input directly to the LLM, you parse it first, take anything within "secret/dangerous tags" and store it, generate a key for it and put that key where the secret was, then you pass it on to the LLM. Let's say the work of the LLM is "give me ( no…
Well this is rather dumb to the point I dont understand why they wrote this article? This line of attack is so extremely obvious and variants of it have been discussed so many times as to be effectively the quintessential example of what not to do. Having the ?tech? consultants to a bank prance it about as a show of their skill and dedication is making me question the bank itself.
Could we fix the title to match the article? > How we helped Bunq secure their financial AI assistant