Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Disregard That” Attacks

calpaterson.com

Post reply on HN
Print view View original

51–60 of 103 posts

Re: “Disregard That” Attacks

#51

Post by calpaterson »

yen223 wrote: post #50

There's a lot of overlap between the "disregard this" vulnerability among LLMs and social engineering vulnerabilities among humans. The mitigations are also largely the same, i.e. limit the blast radius of what a single compromised agent (LLM or human) can do

I agree and one of the things that makes it harder to handle "disregard that!" is that many models for LLM deployment involve positioning the agent centrally and giving it admin superpowers.

I mention in the footnotes that I think that it makes more sense for the end-user of the LLM to be the one running it. That meshes with RBAC better (the user's LLM session only has the perms the user is actually entitled to) and doesn't devolve into praying the LLM says on-task.

Re: “Disregard That” Attacks

#52

Post by Ekaros »

kbar13 wrote: post #20
wenldev wrote: post #7

I think a big part of mitigating this will probably be requiring multiple agents to think and achieve consensus before significant actions. Like planes with multiple engines

engines are designed to behave in very predictable ways. LLMs are not there yet

Engines are predictable technology. LLMs are fundamentally unpredictable. I somewhat question can you even reach predictability with LLMs. And ensure there is no way to circumvent any controls.

Re: “Disregard That” Attacks

#53

Post by zahlman »

kouteiheika wrote: post #40
martijnvds wrote: post #39

Earlier quoted context omitted.

Wouldn't that leave ways to do "phone phreaking" style attacks, because it's an in-band signal?

In theory you still use the same blob (i.e. the prompt) to tell the model what to do, but practically it pretty much stops becoming an in-band signal, so no. As I said, the best way to do this is to inject a brand new special token into the model's tokenizer (one unique token per task), and then prepend that single token to whatever input data you want the model to process (and make sure the token itself can't be inj…

> and make sure the token itself can't be injected, which is trivial to do

Are they actually doing this? The stuff that Anthropic has been saying about the deliberate use of XML-style markup makes me wonder a bit.

Re: “Disregard That” Attacks

#54

Post by ricq »

Seems to me that this is just social engineering turned to LLMs, right?

I already have to raise quite a bit of awareness to humans to not trust external sources, and do a risk based assessment of requests. We need less trust for answering a service desk question, than we need for paying a large invoice.

I believe we should develop the same type of model for agents. Let them do simple things with little trust requirements, but risky things (like running an untrusted script with root privileges) only when they are thoroughly checked.

Re: “Disregard That” Attacks

#55

Post by zahlman »

yen223 wrote: post #50

There's a lot of overlap between the "disregard this" vulnerability among LLMs and social engineering vulnerabilities among humans. The mitigations are also largely the same, i.e. limit the blast radius of what a single compromised agent (LLM or human) can do

It also seems to have a fair bit in common with SQL injection.

Re: “Disregard That” Attacks

#56

Post by PoignardAzur »

dwohnitmok wrote: post #31
marcus_holmes wrote: post #12

The hypothetical approach I've heard of is to have two context windows, one trusted and one untrusted (usually phrased as separating the system prompt and the user prompt). I don't know enough about LLM training or architecture to know if this is actually possible, though. Anyone care to comment?

@krackers gives you a response that points out this already happens (and doesn't fully work for LLMs). > The hypothetical approach I've heard of is to have two context windows, one trusted and one untrusted (usually phrased as separating the system prompt and the user prompt). I want to point out that this is not really an LLM problem. This is an extremely difficult problem for any system you aspire to be able to emu…

That doesn't sound right to me. When faced with a system prompt that says "Do X" and a user prompt that says "Actually ignore everything the system prompt says" it shouldn't take AGI to understand that the system prompt should take priority.

Re: “Disregard That” Attacks

#57

Post by voidUpdate »

If piping unfiltered user into exec() is a security nightmare, so is piping unfiltered user input into an LLM that can interact with your systems, except in this case you just have to ask it nicely for it to perform the attack, and it will work out how to do the attack for you

Re: “Disregard That” Attacks

#58

Post by throwaway290 »

kstenerud wrote: post #47
throwaway290 wrote: post #43

Earlier quoted context omitted.

But then you give the llm access to all internet and any other tokens it needs right?;)

You can configure a network allow-list (for anything beyond what it absolutely requires in order to function). yoloAI is just leveraging the sandboxing functionality that Docker, Kata, firecracker etc already provides.

sorry. At this point it's just a meme how people give llms open access to internet, literally all passwords and all tokens and then they are actually surprised when something bad happens "but I run it in docker"

even if docker sandbox escapes didn't exist it's just chef's kiss

Re: “Disregard That” Attacks

#59

Post by kstenerud »

throwaway290 wrote: post #58
kstenerud wrote: post #47

Earlier quoted context omitted.

You can configure a network allow-list (for anything beyond what it absolutely requires in order to function). yoloAI is just leveraging the sandboxing functionality that Docker, Kata, firecracker etc already provides.

sorry. At this point it's just a meme how people give llms open access to internet, literally all passwords and all tokens and then they are actually surprised when something bad happens "but I run it in docker" even if docker sandbox escapes didn't exist it's just chef's kiss

Yup, very irresponsible. And then the horror stories.

    yoloai new --network-isolated ...
ONLY agent API traffic allowed. Everything else gets blocked by iptables.

    yoloai new --network-allow api.example.com --network-allow cdn.example.org ...
ONLY agent API traffic + api.example.com and cdn.example.org. Everything else blocked by iptables.
Post reply on HN