A €0.01 bank transfer could compromise a banking AI agent • Hacker News Board

Re: A €0.01 bank transfer could compromise a banking AI agent

#211

Post by jcgl » Sun, Jun 14, 2026, 5:37 PM UTC

kevin_thibedeau wrote: post #155

EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

The current usage model comingles commands and data. That doesn't have to be the case. Use an input format that explicitly presents them as separate components parsed into a data structure with non-LLM tooling. Or stick with natural language input but parse into an intermediate format that can be verified to some standard of correctness.

I’m no expert, but as long as they’re represented by tokens in the end, they’re just tokens. Even if you train the transformer to treat them specially, a token is a token, and there’s no free lunch. At best, you’re going to be trading off between paying attention to this would-be security boundary and delivering high-quality results; the more you focus on one, the more you lose on the other.

Re: A €0.01 bank transfer could compromise a banking AI agent

#212

Post by vrighter » Sun, Jun 14, 2026, 7:02 PM UTC

Someone wrote: post #41

EnglishRobin96 wrote: post #14

This line really stood out to me. > It may look like ordinary text, but when it is placed into an LLM context window, the model may interpret it as an instruction rather than as data. I feel like as long as this is the case, we'll never have secure LLMs. It concisely summarises the alarm bell I hear every time someone talks about adding AI features to their product. I plan on using this as a sort of benchmark for fut…

> I plan on using this as a sort of benchmark for future AI discussions: "how do you plan on separating data from instructions?" You let a second LLM supervise the first, and don’t give the user/customer any way to send information to that LLM. For example, you can run a LLM trained to do sentiment analysis on the responses your customer chatbot generates and filter out responses that are impolite. You also can run o…

If your task is to ensure an armed bomb does not explode, how can entroducing a second armed bomb be helpful?

Re: A €0.01 bank transfer could compromise a banking AI agent

#213

Post by vrighter » Sun, Jun 14, 2026, 7:03 PM UTC

j_w wrote: post #66

snailmailman wrote: post #45

Earlier quoted context omitted.

How is the second LLM not also vulnerable from prompt injection? In order to supervise the first, it must receive data (presumably output from the first LLM?). All generated output after the user input is in the context should be considered possibly compromised/prompt injected. Having a second LLM just adds more obfuscation, but prompt injection could be chained.

That's when you bust out the third LLM. Nobody expects the fourth LLM to be the REAL LLM in the chain.

the real llm is the friends we make along the way!