11–20 of 103 posts
It did get me thinking the extent to which I could bypass the original prompt and use someone else's tokens for free.
I don't know enough about LLM training or architecture to know if this is actually possible, though. Anyone care to comment?
The bowdlerisation of today's internet continues to annoy me. To be clear, the joke is traditionally "HAHA DISREGARD THAT, I SUCK COCKS".
I didn’t see the article talk specifically about this, or at least not in enough detail, but isn’t the de-facto standard mitigation for this to use guardrails which lets some other LLM that has been specifically tuned for these kind of things evaluate the safety of the content to be injected? There are a lot of services out there that offer these types of AI guardrails, and it doesn’t have to be expensive. Not saying…
The article does mention this and a weakness of that approach is mentioned too.
The hypothetical approach I've heard of is to have two context windows, one trusted and one untrusted (usually phrased as separating the system prompt and the user prompt). I don't know enough about LLM training or architecture to know if this is actually possible, though. Anyone care to comment?
The bowdlerisation of today's internet continues to annoy me. To be clear, the joke is traditionally "HAHA DISREGARD THAT, I SUCK COCKS".
The bowdlerisation of today's internet continues to annoy me. To be clear, the joke is traditionally "HAHA DISREGARD THAT, I SUCK COCKS".
The bowdlerisation of today's internet continues to annoy me. To be clear, the joke is traditionally "HAHA DISREGARD THAT, I SUCK COCKS".
But that has bad words in it! EDIT: https://web.archive.org/web/20080702204110/http://bash.org/?...
The hypothetical approach I've heard of is to have two context windows, one trusted and one untrusted (usually phrased as separating the system prompt and the user prompt). I don't know enough about LLM training or architecture to know if this is actually possible, though. Anyone care to comment?
The problem is that if information can flow from the untrusted window to the trusted window then information can flow from the untrusted window to the trusted window. It's like https://textslashplain.com/2017/01/14/the-line-of-death/ except there isn't even a line in the first place, just the fuzzy point where you run out of context.
The distinction I think this idea includes is that the distinction between contexts is encoded into the training or architecture of the LLM. So (as I understand it) if there is any conflict between what's in the trusted context and the untrusted context, then the trusted context wins. In effect, the untrusted context cannot just say "Disregard that" about things in the trusted context.
This obviously means that there can be no flow of information (or tokens) from the untrusted context to the trusted context; effectively the trusted context is immutable from the start of the session, and all new data can only affect the untrusted context.
However, (as I understand it) this is impossible with current LLM architecture because it just sees a single stream of tokens.
I think a big part of mitigating this will probably be requiring multiple agents to think and achieve consensus before significant actions. Like planes with multiple engines