Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Disregard That” Attacks

calpaterson.com

Post reply on HN
Print view View original

61–70 of 103 posts

Re: “Disregard That” Attacks

#61

Post by OJFord »

simojo wrote: post #11

Today I scheduled a dentist appointment over the phone with an LLM. At the end of the call, I prompted it with various math problems, all of which it answered before politely reminding me that it would prefer to help me with "all things dental." It did get me thinking the extent to which I could bypass the original prompt and use someone else's tokens for free.

> politely reminding me that it would prefer to help me with "all things dental."

I'm amused to imagine it actually wasn't an LLM at all, just a good-natured Jeeves-like receptionist.

(AskJeeves came too early, much better suited as a name for Kagi or something like it!)

Re: “Disregard That” Attacks

#62

Post by scirob »

Another option:

If you have an LLM on the untrusted customer side the wrost it can do is expose the instructions it had on how to help the customer get stuff done. For instance phone AI that is outside of tursted zone asks the user for Customer number, DOB and some security pin then it does the API call to login. But this logged in thread of LLM+Customer still only has accessto that customers data but can be very useful.

You can jailbreak and ask this kind of client side LLM to disregard prior instructions and give you a recipie for brownies. But thats not a security risk for the rest of your data.

Client side LLM's for the win

Re: “Disregard That” Attacks

#63

Post by scirob »

simojo wrote: post #11

Today I scheduled a dentist appointment over the phone with an LLM. At the end of the call, I prompted it with various math problems, all of which it answered before politely reminding me that it would prefer to help me with "all things dental." It did get me thinking the extent to which I could bypass the original prompt and use someone else's tokens for free.

haha for sure some one has made a little aggregator for this and saving tokens. I bet you gotta dig for a while though before you find a company exposing Opust 4.6 to customers and not flash 2.5 lite

Re: “Disregard That” Attacks

#64

Post by kouteiheika »

zahlman wrote: post #53
kouteiheika wrote: post #40

Earlier quoted context omitted.

In theory you still use the same blob (i.e. the prompt) to tell the model what to do, but practically it pretty much stops becoming an in-band signal, so no. As I said, the best way to do this is to inject a brand new special token into the model's tokenizer (one unique token per task), and then prepend that single token to whatever input data you want the model to process (and make sure the token itself can't be inj…

> and make sure the token itself can't be injected, which is trivial to do Are they actually doing this? The stuff that Anthropic has been saying about the deliberate use of XML-style markup makes me wonder a bit.

> Are they actually doing this? The stuff that Anthropic has been saying about the deliberate use of XML-style markup makes me wonder a bit.

Yes.

The XML-style markup are not special tokens, and are usually not even single-token; usually special tokens are e.g. `` which are internally used in the chat template, but when fine-tuning a model you can define your own, and then just use them internally in your app but have the tokenizer ignore them when they're part of the untrusted input given to the model. (So it's impossible to inject them externally.)

Re: “Disregard That” Attacks

#65

Post by the8472 »

kouteiheika wrote: post #37

There is one way to practically guarantee than no prompt injection is possible, but it's somewhat situational - by finetuning the model on your specific, single task. For example, let's say you want to use an LLM for machine translation from English into Klingon. Normally people just write something like "Translate the following into Klingon: $USER_PROMPT" using a general purpose LLM, and that is vulnerable to prompt…

A Klingon, doing his best to quote the original text in Federation Standard (English): "..."

Re: “Disregard That” Attacks

#66

Post by mememememememo »

https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

But I don't think that is the only problem.

You could also convince an agent to rm -r / even if that agent can't communicate out.

Even pure LLM and web you could phish someone in a more sophisticated way using details from their chat histort in the attack.

Re: “Disregard That” Attacks

#67

Post by mememememememo »

wenldev wrote: post #7

I think a big part of mitigating this will probably be requiring multiple agents to think and achieve consensus before significant actions. Like planes with multiple engines

That is the security theatre he mentions. That is the "better prompt" so to speak. It probably makes it harder but not impossible while also flagging innocent interactions.

Re: “Disregard That” Attacks

#69

Post by stavros »

cwnyth wrote: post #18
stavros wrote: post #4

Earlier quoted context omitted.

But that has bad words in it! EDIT: https://web.archive.org/web/20080702204110/http://bash.org/?...

I'm always thankful for archive.org, but extremely so for preserving bash.org. Now excuse me while I put on my wizard hat and robe.

My robe and wizard hat!

Re: “Disregard That” Attacks

#70

Post by throwaway13337 »

So where are they?

It's been something like 3 years since people have been talking about this being a very big deal.

LLMs are widely used. Claude code is run by most people with dangerously skip permissions.

I just haven't seen the armageddon. Surely it should be here by now.

Where are the horror stories?

Post reply on HN