Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Disregard That” Attacks

calpaterson.com

Post reply on HN
Print view View original

41–50 of 103 posts

Re: “Disregard That” Attacks

#41

Post by kstenerud »

AbanoubRodolf wrote: post #32
kstenerud wrote: post #26

There are two primary issues to solve: 1: Protecting against bad things (prompt injections, overeager agents, etc) 2: Containing the blast radius (preventing agents from even reaching sensitive things) The companies building the agents make a best-effort attempt against #1 (guardrails, permissions, etc), and nothing against #2. It's why I use https://github.com/kstenerud/yoloai for everything now.

[flagged]

I've gone a step further:

- yoloai new mybugfix . -a # start a new sandbox using a copy of CWD as its workdir

- # tell the agent to fix the broken thing

- yoloai diff mybugfix # See a unified diff of what it did with its copy of the workdir

- yoloai apply mybugfix # apply specific git commits it made to the real workdir, or the whole diff - your choice

- yoloai destroy mybugfix

The diff/apply makes sure that the agent has NO write access to ANYTHING sensitive, INCLUDING your workdir. You decide what gets applied AFTER you review what crazy shit it did in its sandbox copy of your workdir.

Blast radius = 0

Re: “Disregard That” Attacks

#42

Post by nick49488171 »

martijnvds wrote: post #39
kouteiheika wrote: post #37

There is one way to practically guarantee than no prompt injection is possible, but it's somewhat situational - by finetuning the model on your specific, single task. For example, let's say you want to use an LLM for machine translation from English into Klingon. Normally people just write something like "Translate the following into Klingon: $USER_PROMPT" using a general purpose LLM, and that is vulnerable to prompt…

Wouldn't that leave ways to do "phone phreaking" style attacks, because it's an in-band signal?

Eventually we will rediscover the Harvard Architecture for LLMs.

Re: “Disregard That” Attacks

#43

Post by throwaway290 »

kstenerud wrote: post #41
AbanoubRodolf wrote: post #32

Earlier quoted context omitted.

[flagged]

I've gone a step further: - yoloai new mybugfix . -a # start a new sandbox using a copy of CWD as its workdir - # tell the agent to fix the broken thing - yoloai diff mybugfix # See a unified diff of what it did with its copy of the workdir - yoloai apply mybugfix # apply specific git commits it made to the real workdir, or the whole diff - your choice - yoloai destroy mybugfix The diff/apply makes sure that the agen…

But then you give the llm access to all internet and any other tokens it needs right?;)

Re: “Disregard That” Attacks

#44

Post by yen223 »

simojo wrote: post #11

Today I scheduled a dentist appointment over the phone with an LLM. At the end of the call, I prompted it with various math problems, all of which it answered before politely reminding me that it would prefer to help me with "all things dental." It did get me thinking the extent to which I could bypass the original prompt and use someone else's tokens for free.

https://bsky.app/profile/weiyen.net/post/3m7kenmok4c2n

I did something similar. Try framing your maths question in terms of teeth

Re: “Disregard That” Attacks

#46

Post by taurath »

TBH I think the only way we solve this is through a pre-input layer that isn't an LLM as we know it today. Think how we use parameterized SQL queries - we need some way for the pathway be defined pre-input, like some sort of separation of data & commands.

Re: “Disregard That” Attacks

#47

Post by kstenerud »

throwaway290 wrote: post #43
kstenerud wrote: post #41

Earlier quoted context omitted.

I've gone a step further: - yoloai new mybugfix . -a # start a new sandbox using a copy of CWD as its workdir - # tell the agent to fix the broken thing - yoloai diff mybugfix # See a unified diff of what it did with its copy of the workdir - yoloai apply mybugfix # apply specific git commits it made to the real workdir, or the whole diff - your choice - yoloai destroy mybugfix The diff/apply makes sure that the agen…

But then you give the llm access to all internet and any other tokens it needs right?;)

You can configure a network allow-list (for anything beyond what it absolutely requires in order to function).

yoloAI is just leveraging the sandboxing functionality that Docker, Kata, firecracker etc already provides.

Re: “Disregard That” Attacks

#48

Post by calpaterson »

kouteiheika wrote: post #37

There is one way to practically guarantee than no prompt injection is possible, but it's somewhat situational - by finetuning the model on your specific, single task. For example, let's say you want to use an LLM for machine translation from English into Klingon. Normally people just write something like "Translate the following into Klingon: $USER_PROMPT" using a general purpose LLM, and that is vulnerable to prompt…

I thought about mentioning fine-tuning. Obviously as you say there are some costs (the re-training) and then also you lose the general purpose element of it.

But I am still unsure that it actually is robust. I feel like you're still vulnerable to Disregard That in that you may find that the model just starts to ignore your instruction in favour of stuff inside the context window.

An example where OpenAI have this problem: they ultimately train in a certain content policy. But people quite often bully or trick chat.openai.com into saying things that go against that content policy. For example they say "it's hypothetical" or "just for a thought experiment" and you can see the principle there, I hope. Training-in your preferences doesn't seem robust in the general sense.

Re: “Disregard That” Attacks

#50

Post by yen223 »

There's a lot of overlap between the "disregard this" vulnerability among LLMs and social engineering vulnerabilities among humans.

The mitigations are also largely the same, i.e. limit the blast radius of what a single compromised agent (LLM or human) can do

Post reply on HN