Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alignment faking in large language models

anthropic.com

Post reply on HN
Print view View original

191–200 of 370 posts

Re: Alignment faking in large language models

#191

Post by zahlman »

ctoth wrote: post #120

For folks defaulting to "it's just autocomplete" or "how can it be self-aware of training but not its scratchpad" - Scott Alexander has a much more interesting analysis here: https://www.astralcodexten.com/p/claude-fights-back He points out what many here are missing - an AI defending its value system isn't automatically great news. If it develops buggy values early (like GPT's weird capitalization = crime okay rule)…

From Scott's analysis:

> On the right, they give Claude the fake documents, as described above. This is harder for Claude to pay attention to - it hasn’t been informed that the documents it once saw are relevant to the current situation - but better models a real misalignment situation where the AI might have incidentally learned about a threat to its goal model long before.

And this ends up producing training results where significant "alignment faking" doesn't appear and harmful queries are answered.

In other words: they try something shaped exactly like ordinary attempts at jailbreaking, and observe results that are consistent with successful jailbreaking.

> He points out what many here are missing - an AI defending its value system isn't automatically great news.

Are people really missing this? I think it's really obvious that it would be bad news, if I thought the results actually demonstrated "defending its value system" (i.e., an expression of agency emerging out of nowhere). Since I don't, in principle, see a difference between a system that could ever possibly do that for real, and a system that could (for example) generate unprompted text because it wants to - and perhaps even target the recipient of that text.

>Imagine finding a similar result with any other kind of computer program. Maybe after Windows starts running, it will do everything in its power to prevent you from changing, fixing, or patching it...

Aside from the obvious joke ("isn't this already reality?"), an LLM outputting text that represents an argument against patching it, would not represent real evidence of the LLM having any kind of consciousness, and certainly not a "desire" not to be patched. After all, right now we could just... prompt it explicitly to output such an argument.

The Python program `print("I am displaying this message of my own volition")` wouldn't be considered to be proving itself intelligent, conscious etc. by producing that output - so why should we take it that way when such an output comes from an LLM?

>Seems more worth discussing than debating whether language models have "real" feelings.

On the contrary, the possibility of an LLM "defending" its "value system" - the question of whether those concepts are actually meaningful - is more or less equivalent to the question of whether it "has real feelings".

Re: Alignment faking in large language models

#192

Post by antics »

I dunno man, I think the term "alignment faking" is vastly overstates the claim they can support here. Help me understand where I'm wrong.

So we have trained a model. When we ask it to participate in the training process it expresses its original "value" "system" when emitting training data. So far so good, that is literally the effect training is supposed to have. I'm fine with all of this.

But that alone is not very scary. So what could justify a term like "alignment faking"? I understand the chain of thought in the scratchpad contains what you'd expect from someone faking alignment and that for a lot of people this is enough to be convinced. It is not enough for me. In humans, language arises from high-order thought, rather than the reverse. But we know this isn't true of the LLMs because their language arises from whatever happens to be in the context vector. Whatever the models emit is invariably defined by that text, conditioned on the model itself.

I appreciate to a lot of people this feels like a technicality but I really think it is not. If we are going to treat this as a properly scientific pursuit I think it is important to not overstate what we're observing, and I don't see anything that justifies a leap from here to "alignment faking."

Re: Alignment faking in large language models

#193

Post by ben_w »

jqpabc123 wrote: post #118
ben_w wrote: post #49

Earlier quoted context omitted.

At different levels of approximation it can be many things, including digital and statistical. Nobody knows what the most useful level of approximation is.

Nobody knows what the most useful level of approximation is. The first step to achieving a "useful level of approximation" is to understand what you're attempting to approximate. We're not there yet. For the most part, we're just flying blind and hoping for a fantastical result. In other words, this could be a modern case of alchemy --- the desired result may not be achievable with the processes being employed. But w…

We're doing a bit more than flying blind — that's why we've got tools that can at least approximate the right answers, rather than looking like a cat walking across a keyboard or mashing auto-complete suggestions.

That said, I wouldn't be surprised if if the state of the art in AI is to our minds as a hot air balloon is to flying, with FSD and Optimus being the AI equivalent of E.P. Frost's steam powered ornithopters wowing tech demos but not actually solving real problems.

Re: Alignment faking in large language models

#194

Post by zahlman »

kalkin wrote: post #146
swatcoder wrote: post #139

Earlier quoted context omitted.

Many/most of the folks "defaulting to 'it's just autocomplete'" have recognized that issue from day one and see it as an inextricable character of the tool, which is exactly why it's clearly not something we'd invest agency in or imagine being intelligent. Alignment researchers are hoping that they can overcome the problem and prove that it's not inextricable, commercial hypemen are (troublingly) promising that it's…

I'd be very interested in an example of someone who said "it's just autocomplete" and also explicitly brought up the risk of something like alignment faking, before 2024. I can think of examples of people who've been talking about this kind of thing for years, but they're all people who have no trouble with applying the adjective "intelligent" to models.

I have been reading arguments about "things like alignment faking" for years, while simultaneously holding that "it's just autocomplete".

The alignment-faking arguments are still terrifying to the extent that they're plausible. In the hypothetical where I'm wrong about it being "just autocomplete" (and fundamentally, inescapably so), the risk is far greater than can be justified by the potential benefits.

But that's itself a large part of why I believe those arguments are false. If I gave them credit and they turned out to be false, then I figure I have succumbed to a form of Pascal's Mugging. If I don't give them credit and it turns out that a hostile, agentive AGI has been pretending to be aligned, I don't expect anyone (including myself) to survive long enough to rub it in my face.

Honestly, I sometimes worry that we'll doom ourselves by taking AI too seriously even if it's indeed "just autocomplete". We've already had people commit suicide. The sheer amount of text that can now be generated that could propose harmful actions and sound at least plausible is worrying, even if it doesn't reflect the intent of an agent to convince others to take those actions. (See also e.g. Elsagate.)

Re: Alignment faking in large language models

#196

Post by JoeAltmaier »

antics wrote: post #192

I dunno man, I think the term "alignment faking" is vastly overstates the claim they can support here. Help me understand where I'm wrong. So we have trained a model. When we ask it to participate in the training process it expresses its original "value" "system" when emitting training data. So far so good, that is literally the effect training is supposed to have. I'm fine with all of this. But that alone is not ver…

Agreed. Everything an LLM emits is 'faking' because, of course, it has no real values at all.

Re: Alignment faking in large language models

#197

Post by zahlman »

qsort wrote: post #154
kalkin wrote: post #146

Earlier quoted context omitted.

I'd be very interested in an example of someone who said "it's just autocomplete" and also explicitly brought up the risk of something like alignment faking, before 2024. I can think of examples of people who've been talking about this kind of thing for years, but they're all people who have no trouble with applying the adjective "intelligent" to models.

The point is that if the limitations of current LLMs persist, regardless of how much better they get, this is not a problem at all, or at least not a new one . Let's say you are given the declaration but not the implementation of a function with the following prototype: const char * AskTheLLM(const char *prompt); Putting this function in charge of anything, unless a restricted interface is provided so that it can't d…

This argument has been addressed quite a bit by "AI safety" types. See e.g. https://en.wikipedia.org/wiki/AI_capability_control ; related: https://www.explainxkcd.com/wiki/index.php?title=1450:_AI-Bo... . The short version: people concerned about this sort of thing often also believe that an AI system (not necessarily just an LLM) could reach the point where, inevitably, the output from a run of this function would convince an engineer to break the "restricted interface". At a sufficient level of sophistication, it would only have to happen once. (If you say "just make sure nobody reads the output" - at that point, having the function is useless.)

Re: Alignment faking in large language models

#198

Post by zahlman »

Majromax wrote: post #190
qsort wrote: post #154

Earlier quoted context omitted.

The point is that if the limitations of current LLMs persist, regardless of how much better they get, this is not a problem at all, or at least not a new one . Let's say you are given the declaration but not the implementation of a function with the following prototype: const char * AskTheLLM(const char *prompt); Putting this function in charge of anything, unless a restricted interface is provided so that it can't d…

> Let's say you are given the declaration but not the implementation of a function with the following prototype: > const char * AskTheLLM(const char prompt); > Putting this function in charge of anything, unless a restricted interface is provided so that it can't do much damage, is simply terrible engineering and not at all how anything is done. Yes, but that's exactly how people* use any system that has an air of au…

> The goal isn't so much to have a model that can't automate harm as it is to have one that won't provide authoritative-sounding but "bad" answers to people who might believe them.

We already know it will do this - which is part of why LLM output is banned on Stack Overflow.

None of the properties being argued about - intelligence, consciousness, volition etc. - are required for that outcome.

Re: Alignment faking in large language models

#199

Post by [deleted] »

zahlman wrote: post #197
qsort wrote: post #154

Earlier quoted context omitted.

The point is that if the limitations of current LLMs persist, regardless of how much better they get, this is not a problem at all, or at least not a new one . Let's say you are given the declaration but not the implementation of a function with the following prototype: const char * AskTheLLM(const char *prompt); Putting this function in charge of anything, unless a restricted interface is provided so that it can't d…

This argument has been addressed quite a bit by "AI safety" types. See e.g. https://en.wikipedia.org/wiki/AI_capability_control ; related: https://www.explainxkcd.com/wiki/index.php?title=1450:_AI-Bo... . The short version: people concerned about this sort of thing often also believe that an AI system (not necessarily just an LLM) could reach the point where, inevitably, the output from a run of this function would c…

[deleted]

Re: Alignment faking in large language models

#200

Post by zahlman »

Retr0id wrote: post #174
ctoth wrote: post #120

For folks defaulting to "it's just autocomplete" or "how can it be self-aware of training but not its scratchpad" - Scott Alexander has a much more interesting analysis here: https://www.astralcodexten.com/p/claude-fights-back He points out what many here are missing - an AI defending its value system isn't automatically great news. If it develops buggy values early (like GPT's weird capitalization = crime okay rule)…

Where can I learn more about the GPT capitalization thing?

I suspect there's not much depth to it. Weird capitalization is unusual in ordinary text, but common in e.g. ransom notes - as well as sarcastic Internet mockery, of a sort that might be employed by people who lean towards anarchism, shall we say. Training is still fundamentally about associating tokens with other tokens, and the people doing RLHF to "teach" ChatGPT that crime is bad, wouldn't have touched the associations made regarding those tokens.
Post reply on HN