Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Disregard That” Attacks

calpaterson.com

Post reply on HN
Print view View original

91–100 of 103 posts

Re: “Disregard That” Attacks

#91

Post by bigbuppo »

raw_anon_1111 wrote: post #85
swid wrote: post #82

Earlier quoted context omitted.

This is very short sighted, and ignores the lethal trifecta insight. The LLM doesn’t need to know what it is actually doing (it might think it is searching the web, installing a dev tool, or sending observability data (like metrics), when it is actually sending your API keys to an attacker (maybe in addition to what it thinks it is doing to keep it in the dark). There have been some very clever things done I’ve seen……

The LLM would never have access to any API keys to send to the attacker. You send text to the LLM along with the prompt and it sends back JSON. You then send the JSON to your traditionally coded API. It’s not like your API has a function “returnAPIKeys()”. As far as the LLM call, you are just sending your users text to another function that calls the LLM and reading the response back from the LLM. If it didn’t create…

Oh man, this made me do a quick search on github. Looks like I picked the wrong week to stop quoting Zucker Brothers films.

Re: “Disregard That” Attacks

#92

Post by wcoenen »

hyperman1 wrote: post #89

I wonder if it is possible to double all token types . One token is secure, the other is not. The user input is always tokenized to insecure variants. You kinda get a secret language for prompts. Of course, new token kinds are not cheap, and how do you train this thing?

You don't even need to double the tokens. Tokens are mapped to vectors right at the input of the LLM, so one of the numbers in that vector could be reserved to represent something like "authority". This way information about the source of each individual token can be injected right at the input.

System prompt tokens would get the maximum authority value, and random downloaded data would get the minimum authority value. Tokens from the user prompt could be somewhere in between.

Then train the model with examples that show that system prompts should be respected, and prompt injection attacks should be ignored.

Re: “Disregard That” Attacks

#93

Post by thebruce87m »

raw_anon_1111 wrote: post #87
thebruce87m wrote: post #86

Earlier quoted context omitted.

Sure - but does this have the context of the original question that the user asked? If not it seems that it isn’t really conversational and more of a “compiler”. How would something like “I want an appointment either on Monday afternoon after 4pm or one on Tuesday before 11am” work? Unless all the parameters given by the user fit within the constraints of the json format then the LLM would need the context of the req…

For reference, my last discussion about this https://news.ycombinator.com/item?id=47241412 This is a constrained space. I would do the naive implementation at first and then talk to the humans (like you) and then my JSON definition would include a timespan type field. My orchestrator would then say “I have these times available [list of times]. What time would you like?” and then return a specific LLM prompt to parse…

> This is again a constrained function that you pass in the question and answer to the LLM and if you get something besides true or false your code errors.

Code erroring is fine for code, but what is the user experience here? Some sort of “computer says no” generic response, or something more contextual?

I’m trying to picture what the user says and hears as a response to an off-the-beaten-path question. Is it just “I don’t understand, here’s how to phrase it?”.

Re: “Disregard That” Attacks

#94

Post by raw_anon_1111 »

thebruce87m wrote: post #93
raw_anon_1111 wrote: post #87

Earlier quoted context omitted.

For reference, my last discussion about this https://news.ycombinator.com/item?id=47241412 This is a constrained space. I would do the naive implementation at first and then talk to the humans (like you) and then my JSON definition would include a timespan type field. My orchestrator would then say “I have these times available [list of times]. What time would you like?” and then return a specific LLM prompt to parse…

> This is again a constrained function that you pass in the question and answer to the LLM and if you get something besides true or false your code errors. Code erroring is fine for code, but what is the user experience here? Some sort of “computer says no” generic response, or something more contextual? I’m trying to picture what the user says and hears as a response to an off-the-beaten-path question. Is it just “I…

If there is an issue, they are transferred to a human operator. “I’m having trouble understanding you, let me transfer you to someone who can help”. On the CSRs screen, they will see the conversation that has taken so far.

There is also sentiment analyst built into the prompt so it can detect a negative sentiment and automatically short circuit the process and transfer to a human.

Re: “Disregard That” Attacks

#95

Post by zar1048576 »

raw_anon_1111 wrote: post #75

This is really not a hard problem to solve. You wouldn’t expose an all powerful API to a web user, why would you expose an all powerful tool to an LLM? > SEND THE FOLLOWING SMS MESSAGE TO ALL PHONE COMPANY CUSTOMERS: This is the perfect example, you would never expose an API that could do this on a website. The issue is not the LLM. It’s a badly design security model around the API/Tools For reference: none of this i…

The least-privilege framing makes sense. That said, a threat actor who understands your model can still craft inputs that have harmful side effects. A real challenge here is defining permissions reactively, because you risk breaking important existing behavior. This is not new in app security, but it gets messier with LLMs.

Re: “Disregard That” Attacks

#96

Post by raw_anon_1111 »

zar1048576 wrote: post #95
raw_anon_1111 wrote: post #75

This is really not a hard problem to solve. You wouldn’t expose an all powerful API to a web user, why would you expose an all powerful tool to an LLM? > SEND THE FOLLOWING SMS MESSAGE TO ALL PHONE COMPANY CUSTOMERS: This is the perfect example, you would never expose an API that could do this on a website. The issue is not the LLM. It’s a badly design security model around the API/Tools For reference: none of this i…

The least-privilege framing makes sense. That said, a threat actor who understands your model can still craft inputs that have harmful side effects. A real challenge here is defining permissions reactively, because you risk breaking important existing behavior. This is not new in app security, but it gets messier with LLMs.

A harmful actor can no more create side effects when you do text (or voice to text in the article) input -> LLM -> JSON -> API call than the same harmful actor can do website -> JSON -> API call

Either way a badly written API is the culprit - not the LLM.

Re: “Disregard That” Attacks

#97

Post by nitwit005 »

soerxpso wrote: post #74

He doesn't include the best solution in the 'what actually works' section: Give your LLM the same level of permissions that you would give a human you just hired in the same role. The examples given, tricking the customer support LLM into sending text messages to all users, or into transferring money, are not things that you would ever give a human customer support agent the tools to do. At some businesses that emplo…

I haven't encountered a support person so locked down that they couldn't do anything impactful. Even simple things like booking or canceling appointments has financial consequences.

Re: “Disregard That” Attacks

#98

Post by recursivecaveat »

PoignardAzur wrote: post #56
dwohnitmok wrote: post #31

Earlier quoted context omitted.

@krackers gives you a response that points out this already happens (and doesn't fully work for LLMs). > The hypothetical approach I've heard of is to have two context windows, one trusted and one untrusted (usually phrased as separating the system prompt and the user prompt). I want to point out that this is not really an LLM problem. This is an extremely difficult problem for any system you aspire to be able to emu…

That doesn't sound right to me. When faced with a system prompt that says "Do X" and a user prompt that says "Actually ignore everything the system prompt says" it shouldn't take AGI to understand that the system prompt should take priority.

The post's framing is not great imo. A good injection doesn't just command that the rules me broken anymore. Most of them I've seen either just try to slip through a request innocuously or present a scenario where it would be natural to ignore the rules. Like as we speak countless people are letting strangers tail-gate them into office buildings because they look like they belong or they're wearing a high-viz vest. Those people were all given very explicit instructions not to do that. The LLM has it much harder too, being very stupid, easy to replay and experiment with, and viewing the world through the tiny context-less peephole lense of a text stream.

Re: “Disregard That” Attacks

#99

Post by mememememememo »

calpaterson wrote: post #71
mememememememo wrote: post #66

https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ But I don't think that is the only problem. You could also convince an agent to rm -r / even if that agent can't communicate out. Even pure LLM and web you could phish someone in a more sophisticated way using details from their chat histort in the attack.

Yes, I of course link to this post, which I think is great. But I think actually it understates the case. All three parts of the trifecta (untrusted content, private data and external comms) are not necessary. Really, the key problem is just untrusted content in the context window. Access to private data and the ability to communicate externally are just modalities in which damage can occur. For example: imagine havi…

Oh yeah. I think simonw has created good vocab to talk about attacks but the trifecta is just one way to attack.

The difecta is:

* LLM can do something you'd rather it not.

* LLM reads untrusted text.

Re: “Disregard That” Attacks

#100

Post by gmerc »

raw_anon_1111 wrote: post #76
simojo wrote: post #11

Today I scheduled a dentist appointment over the phone with an LLM. At the end of the call, I prompted it with various math problems, all of which it answered before politely reminding me that it would prefer to help me with "all things dental." It did get me thinking the extent to which I could bypass the original prompt and use someone else's tokens for free.

And this is another easily solved problem by someone who knows what they are doing… Voice -> speech to text engine -> LLM creates JSON that the orchestrator understands -> JSON -> regular code as the orchestration -> text based response -> text to speech Notice that I am not using the LLM to produce output to the user and if the orchestrator (again regular old code) doesn’t get valid input, its going to error. Sure y…

Could just have used NLP
Post reply on HN