Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Peak LLM?

ihavemanythoughts.substack.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Peak LLM?

#51

Post by qsort »

brookst wrote: post #43
ericb wrote: post #15

I didn't realize until recently is that the "programming" of chatGPT is a hidden prompt fed into the black-box before your document is appended. * ChatGPT's "inability to separate data from code" means every input, even training input, is an eval(). * Is it now impossible to train another LLM on web input? The genie is out of the bottle--you can spam prompts into anything (webforms, html, etc) and compromise future L…

> * ChatGPT's "inability to separate data from code" means every input, even training input, is an eval(). This is very true in GPT3, less true in GPT3.5, and even less true in GPT4. OpenAI is moving to separate system prompts from user prompts. The system prompt is processed first attempts to isolate the user prompt from the system prompt. It's fallible, but getting better. > * LLM's have to be assumed to be entirel…

The thing is that security is binary. One input out of a billion causes bad behavior and you're fucked, exactly like eval, execvpe, sql injections and all their relatives.

The point isn't that you can't use LLM output, it's that you should always consider LLM output as potentially hostile. You can somewhat mitigate this by pairing a LLM with a deterministic system that only allows a predictable subset of behavior, but it's a tricky problem to remove completely.

Re: Peak LLM?

#52

Post by TisButMe »

anyekwest wrote: post #26

What if we just train LLMs to remove prompt injections from inputs? I feel like this isn't an intractable problem.

(author here) How do you know what's a prompt injection vs actual content? If you train another LLM to tell you what's a prompt injection, how do you know it has 100% coverage of all possible injections? OpenAI has been battling people trying to bypass their prompt re-write filter, and as far as I can see, not really winning, just constantly adding stuff to their blocklist until the next thing gets discovered.

Re: Peak LLM?

#53

Post by brookst »

qsort wrote: post #51
brookst wrote: post #43

Earlier quoted context omitted.

> * ChatGPT's "inability to separate data from code" means every input, even training input, is an eval(). This is very true in GPT3, less true in GPT3.5, and even less true in GPT4. OpenAI is moving to separate system prompts from user prompts. The system prompt is processed first attempts to isolate the user prompt from the system prompt. It's fallible, but getting better. > * LLM's have to be assumed to be entirel…

The thing is that security is binary. One input out of a billion causes bad behavior and you're fucked, exactly like eval, execvpe, sql injections and all their relatives. The point isn't that you can't use LLM output, it's that you should always consider LLM output as potentially hostile. You can somewhat mitigate this by pairing a LLM with a deterministic system that only allows a predictable subset of behavior, bu…

> you should always consider LLM output as potentially hostile

Sure, agreed. How is that different from human output?

Re: Peak LLM?

#55

Post by croes »

kolinko wrote: post #45

This issue seems overblown. Sure, if you apply pure GPT-4 (or whatever) to a summarisation task, it will cause the problems mentioned. But you can have another AI that previews content first, looking for prompt injections - and only when the content is deemed safe (or sanitised) it gets forwarded to GPT-4. It's one thing to produce a prompt injection, but another thing to produce prompt injection that avoids detectio…

>But you can have another AI that previews content first, looking for prompt injections

So you can't summarize articles about prompt injections?

Re: Peak LLM?

#56

Post by TisButMe »

m3kw9 wrote: post #25

Just because it’s generated by LLM doesn’t make it crappier than humans. Has anyone did a test if training gpt4 outputs makes it worse? I say gpt4 because this is the one people will unleash in 6 months on max turbo

I'm not too worried about GPTs trained on GPTs, maybe that's an LLM analogy to AlphaGo playing itself a lot to learn how to play go. I'm more worried about people specifically trying to get into the training corpus with biased/wrong/misleading/security-risk content.

Re: Peak LLM?

#57

Post by [deleted] »

billiam wrote: post #14

I have felt this train rumbling down the tracks since GPT-3 hit. He compares peak LLM to what has happened with SEO, but that doesn't really capture it. Gaming the Google algorithm has made the discovery of human-generated content more difficult, but what happens when most of the content to be found by LLM-powered search engines is itself generated by LLMs? The Internet after 2022 rapidly becomes garbage and everythi…

[deleted]

Re: Peak LLM?

#58

Post by qsort »

brookst wrote: post #53
qsort wrote: post #51

Earlier quoted context omitted.

The thing is that security is binary. One input out of a billion causes bad behavior and you're fucked, exactly like eval, execvpe, sql injections and all their relatives. The point isn't that you can't use LLM output, it's that you should always consider LLM output as potentially hostile. You can somewhat mitigate this by pairing a LLM with a deterministic system that only allows a predictable subset of behavior, bu…

> you should always consider LLM output as potentially hostile Sure, agreed. How is that different from human output?

"Human output" isn't automated nor connected to your production systems. Would you let any random user run arbitrary SQL against your production DB?

Re: Peak LLM?

#59

Post by brookst »

ericb wrote: post #44
brookst wrote: post #43

Earlier quoted context omitted.

> * ChatGPT's "inability to separate data from code" means every input, even training input, is an eval(). This is very true in GPT3, less true in GPT3.5, and even less true in GPT4. OpenAI is moving to separate system prompts from user prompts. The system prompt is processed first attempts to isolate the user prompt from the system prompt. It's fallible, but getting better. > * LLM's have to be assumed to be entirel…

> This is very true in GPT3, less true in GPT3.5, and even less true in GPT4. Can you point to evidence that this improvement is the result of something other than a blocklist, because we know blocklists aren't defensible.

I mean it's in the API reference: https://platform.openai.com/docs/guides/chat/introduction

Applications should not use user input for the system role. It's still not a firewall, but it's substantially better than the completion model from GPT3.

There was also a blog post / article / quite somewhere from OpenAI talking about how RL for GPT4 made it treat the system role as more immutable than was true in 3.5, but I'm not finding it in a quick search.

As the technology matures, we'll see security improvements as well. That's kind of the story of tech, right? SQL is doing pretty well despite having a similar problem with instructions versus user data.

I won't hang my hat on LLMs ever being perfect, but nor will I assert they are fundamentally broken and unfixable in this area. It is a very very young technology.

Re: Peak LLM?

#60

Post by TisButMe »

mdale wrote: post #41
TisButMe wrote: post #38

Earlier quoted context omitted.

(Author here) that's what I thought originally, but then it means that LLMs never get to learn from new content - current ones stop in 2021, they don't know that Russia invades Ukraine, or that Arc is a cool browser or the API of any libraries released after their end date (which has been an issue for me for code generation using fast moving libraries). I don't think it's good enough to stop acquiring new content.

There is nothing to prevent a robust hierarchy of rules and training that impacts levels of permissions per operator intent. OpenAi has made a lot of progress on this in a very short amount of time. Casual jailbreaking or negative role playing is already 100x more difficult then early versions via the ChatGPT chat interface. We will see more sophisticated robust adversarial filters to untrusted content going forward.

Possibly yes - I think that's my point with predicting peak oil wrong for 50 years. Still, right now it seems every time OpenAI/someone else adds a new content filter, someone figures out a prompt escape that works.
Post reply on HN