Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A €0.01 bank transfer could compromise a banking AI agent

blue41.com

Post reply on HN
Print view View original

191–200 of 213 posts

Re: A €0.01 bank transfer could compromise a banking AI agent

#191

Post by parliament32 »

tekne wrote: post #121
parliament32 wrote: post #90

Earlier quoted context omitted.

> separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something complete…

I mean: imagine we double our token space to get "red" tokens ans "blue" tokens. Then in all post-training, instructions are red and data is blue. The model can be explicitly trained to ignore instructions written in blue tokens. All external data is blue. All you'd need to do is figure out a nice way to pre-train -- interestingly, you could try pre-training on unfiltered blue data and processed red/blue transcripts!…

Fun schemes like this are all just lipstick on the pig of "asking nicely", unfortunately -- it's just a more creative iteration of "Simon says". It'll improve the probabilities, sure, but you can't guarantee separation like you can in real software. This, like hallucinations, is simply a core facet of LLMs and requires thinking through the threat model and adjusting other parts of the system to accomodate, rather than trying to "solve" IMO.

Re: A €0.01 bank transfer could compromise a banking AI agent

#192

Post by troupo »

Dylan16807 wrote: post #167
troupo wrote: post #136

Earlier quoted context omitted.

> The user asks for details of the last transaction, the user gets back the amount, the source, and the description in a safely quoted format What's "safely quoted format" when prompt injection is already safe in the description? > You can't inject the LLM if it doesn't see the data. How doesn't it see the data when you literally say "The user asks for details of the last transaction, the user gets back the amount, t…

> How doesn't it see the data when you literally say "The user asks for details of the last transaction, the user gets back the amount, the source, and the description"? The above post said how. The LLM writes code to do it. The code has a function to send text to the user. The LLM is not allowed to see the text, only the user is. > And it will make a summary exactly how? The second summarizing-only LLM is fed the ra…

Ah. Now I see you point. This might actually work for a number of situations.

Re: A €0.01 bank transfer could compromise a banking AI agent

#194

Post by nottorp »

EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

> This line really stood out to me.

Where did you people find any text to discuss? I saw like 3 headlines while scrolling past the first 3 screens then gave up.

Re: A €0.01 bank transfer could compromise a banking AI agent

#195

Post by bombcar »

cryo32 wrote: post #28
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

It’s a language model. The spoken and written language we use mixes code and data and requires judgement, experience and intelligence. It’s insanity. We’re fucked.

What if we made SQL injections everything?

Re: A €0.01 bank transfer could compromise a banking AI agent

#200

Post by jimberlage »

parliament32 wrote: post #90
EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

> separating data from instructions There's been a lot of talk about this (for years, honestly), but it all stems from a fundamental nonunderstanding of how LLMs work. There is no distinction for an LLM; "instructions" are a prompt concept, nothing more. It's not possible to separate the two, because LLMs simply take text (ie your instructions, then the data, or maybe in a different order, or maybe something complete…

I would love to have a unicode character for representing the start/end of a data block so that LLMs could at least send data meant to be uncorrupted down a different path at tokenization.
Post reply on HN