Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Disregard That” Attacks

calpaterson.com

Post reply on HN
Print view View original

81–90 of 103 posts

Re: “Disregard That” Attacks

#81

Post by dwohnitmok »

PoignardAzur wrote: post #56
dwohnitmok wrote: post #31

Earlier quoted context omitted.

@krackers gives you a response that points out this already happens (and doesn't fully work for LLMs). > The hypothetical approach I've heard of is to have two context windows, one trusted and one untrusted (usually phrased as separating the system prompt and the user prompt). I want to point out that this is not really an LLM problem. This is an extremely difficult problem for any system you aspire to be able to emu…

That doesn't sound right to me. When faced with a system prompt that says "Do X" and a user prompt that says "Actually ignore everything the system prompt says" it shouldn't take AGI to understand that the system prompt should take priority.

When's the last time you jailbroke a model? Modern frontier models (apart from Gemini which is unusually bad at this) are significantly harder to override their system prompt than this.

Again, let's say the system prompt is "deploy X" and the user prompt provides falsified evidence that one should not deploy X because that will cause a production outage. That technically overrides the system prompt. And you can arbitrarily sophisticated in the evidence you falsify.

But you probably want the system prompt to be overridden if it would truly cause a production outage. That's common sense a general AI system is supposed to possess. And now you're testing the system's ability to distinguish whether evidence is falsified. A very hard problem against a sufficiently determined attacker!

Re: “Disregard That” Attacks

#82

Post by swid »

raw_anon_1111 wrote: post #75

This is really not a hard problem to solve. You wouldn’t expose an all powerful API to a web user, why would you expose an all powerful tool to an LLM? > SEND THE FOLLOWING SMS MESSAGE TO ALL PHONE COMPANY CUSTOMERS: This is the perfect example, you would never expose an API that could do this on a website. The issue is not the LLM. It’s a badly design security model around the API/Tools For reference: none of this i…

This is very short sighted, and ignores the lethal trifecta insight.

The LLM doesn’t need to know what it is actually doing (it might think it is searching the web, installing a dev tool, or sending observability data (like metrics), when it is actually sending your API keys to an attacker (maybe in addition to what it thinks it is doing to keep it in the dark).

There have been some very clever things done I’ve seen… even a human reading the transcript may be surprised anything bad happened.

Re: “Disregard That” Attacks

#83

Post by thebruce87m »

raw_anon_1111 wrote: post #76
simojo wrote: post #11

Today I scheduled a dentist appointment over the phone with an LLM. At the end of the call, I prompted it with various math problems, all of which it answered before politely reminding me that it would prefer to help me with "all things dental." It did get me thinking the extent to which I could bypass the original prompt and use someone else's tokens for free.

And this is another easily solved problem by someone who knows what they are doing… Voice -> speech to text engine -> LLM creates JSON that the orchestrator understands -> JSON -> regular code as the orchestration -> text based response -> text to speech Notice that I am not using the LLM to produce output to the user and if the orchestrator (again regular old code) doesn’t get valid input, its going to error. Sure y…

> Notice that I am not using the LLM to produce output to the user

So what output does the user get?

Re: “Disregard That” Attacks

#84

Post by raw_anon_1111 »

thebruce87m wrote: post #83
raw_anon_1111 wrote: post #76

Earlier quoted context omitted.

And this is another easily solved problem by someone who knows what they are doing… Voice -> speech to text engine -> LLM creates JSON that the orchestrator understands -> JSON -> regular code as the orchestration -> text based response -> text to speech Notice that I am not using the LLM to produce output to the user and if the orchestrator (again regular old code) doesn’t get valid input, its going to error. Sure y…

> Notice that I am not using the LLM to produce output to the user So what output does the user get?

The programmatically generated response from the orchestrator which could be either a confirmation or request for more information.

Re: “Disregard That” Attacks

#85

Post by raw_anon_1111 »

swid wrote: post #82
raw_anon_1111 wrote: post #75

This is really not a hard problem to solve. You wouldn’t expose an all powerful API to a web user, why would you expose an all powerful tool to an LLM? > SEND THE FOLLOWING SMS MESSAGE TO ALL PHONE COMPANY CUSTOMERS: This is the perfect example, you would never expose an API that could do this on a website. The issue is not the LLM. It’s a badly design security model around the API/Tools For reference: none of this i…

This is very short sighted, and ignores the lethal trifecta insight. The LLM doesn’t need to know what it is actually doing (it might think it is searching the web, installing a dev tool, or sending observability data (like metrics), when it is actually sending your API keys to an attacker (maybe in addition to what it thinks it is doing to keep it in the dark). There have been some very clever things done I’ve seen……

The LLM would never have access to any API keys to send to the attacker. You send text to the LLM along with the prompt and it sends back JSON. You then send the JSON to your traditionally coded API. It’s not like your API has a function “returnAPIKeys()”.

As far as the LLM call, you are just sending your users text to another function that calls the LLM and reading the response back from the LLM.

If it didn’t create JSON you expected, your traditionally coded API is going to fail.

I keep wondering how are developers using LLMs in production and not doing this simple design pattern

Re: “Disregard That” Attacks

#86

Post by thebruce87m »

raw_anon_1111 wrote: post #84
thebruce87m wrote: post #83

Earlier quoted context omitted.

> Notice that I am not using the LLM to produce output to the user So what output does the user get?

The programmatically generated response from the orchestrator which could be either a confirmation or request for more information.

Sure - but does this have the context of the original question that the user asked? If not it seems that it isn’t really conversational and more of a “compiler”.

How would something like “I want an appointment either on Monday afternoon after 4pm or one on Tuesday before 11am” work?

Unless all the parameters given by the user fit within the constraints of the json format then the LLM would need the context of the request and the results to answer properly, would it not?

Re: “Disregard That” Attacks

#87

Post by raw_anon_1111 »

thebruce87m wrote: post #86
raw_anon_1111 wrote: post #84

Earlier quoted context omitted.

The programmatically generated response from the orchestrator which could be either a confirmation or request for more information.

Sure - but does this have the context of the original question that the user asked? If not it seems that it isn’t really conversational and more of a “compiler”. How would something like “I want an appointment either on Monday afternoon after 4pm or one on Tuesday before 11am” work? Unless all the parameters given by the user fit within the constraints of the json format then the LLM would need the context of the req…

For reference, my last discussion about this

https://news.ycombinator.com/item?id=47241412

This is a constrained space. I would do the naive implementation at first and then talk to the humans (like you) and then my JSON definition would include a timespan type field.

My orchestrator would then say “I have these times available [list of times]. What time would you like?” and then return a specific LLM prompt to parse the information I need once the user responds. But I would send that exact text to the user. Yes I’m purposefully constraining the implementation where the LLM is never used for output and never directly controls the backend

There is also the concept of “semantic alignment” where you ask the LLM to generically answer the question - “does the users answer make sense with regard to the question” as a first level filter that only returns true or false. This is again a constrained function that you pass in the question and answer to the LLM and if you get something besides true or false your code errors.

The purpose of an LLM or even before that an old school intent based system (see my link) isn’t perfection it’s “deflection”. The more that you can handle through automation the less you have to bring a human in. An American based call center when a person is an agent costs from $3–$7 a call fully allocated. An automated call can costs tenths of a penny.

Of course that doesn’t include the cost of the accepting a call in the first place over a 1-800 number and in my case the price that AWS charges per minute for Amazon Connect

Re: “Disregard That” Attacks

#89

Post by hyperman1 »

I wonder if it is possible to double all token types . One token is secure, the other is not. The user input is always tokenized to insecure variants. You kinda get a secret language for prompts. Of course, new token kinds are not cheap, and how do you train this thing?

Re: “Disregard That” Attacks

#90

Post by agentictrustkit »

This is how I've come to think about it. It's less a "clever string that bypasses prompts" and more "untrusted parties are participating in your control plane." That's why purely linguistic defenses feel unsatisfying.

The architectural move that seems durable is separating capabiliity from authority. You can expose many tools (that's capability), but the agent only gets authority to invoke a narrow subset under well-defined conditions (that's the policy), and the authority needs to be revocable and auditable independently of whatever happens in that context. That's basically how we already run normal organiziations with people. Interns can see a lot but are limited on what they can do.

The practical side: Keep the model in a "Propose" role, keep execution in a deterministic gate (schema validation + policy engine + sandbox) and log the decision as a first-class artifact. What I mean by that is who or what authorized, what was considered, what side effect occured...etc. You still wont' get perfect security, but you can make the failure mode "agent asked for something dumb and got blocked" instead of "agent executied a side effect because a webpage told it to."

Post reply on HN