Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alignment faking in large language models

anthropic.com

Post reply on HN
Print view View original

301–310 of 370 posts

Re: Alignment faking in large language models

#301

Post by antics »

PittleyDunkin wrote: post #286
antics wrote: post #220

Earlier quoted context omitted.

I think what you are saying is that language is deeply and perhaps inextricably tied to human thought. And, I think it's fair to say this is basically uniformly regarded as a fact. The reason I (and others) say that language is almost certainly preceded by (and derived from) high-order thought is because high-order thought exists in all of our close relatives, while language exists only in us. Perhaps the confusion i…

> Perhaps the confusion is in the definition of high-order thought? There is an academic definition but I boil it down to "able to think about thinking as, e.g. all social great apes do when they consider social reactions to their actions." Yes, I think this is it. But now I am confused why "high-order thought" is termed this way when it doesn't include what we would consider "thinking" but rather "cognition". You do…

High-order thought can mean a bunch of things more generally, in this case I meant it to refer to thinking about thinking because "faking" alignment is (I assert) not scary without that.

The reason why is: the core of the paper suggests that they trained a model and then fed it adversarial input, and it mostly (and selectively) kept to its training objectives. This is exactly what we'd expect and want. I think most people will hear that pretty much mostly not be alarmed at all, even lay people. It's only alarming if we say it is "faking" "alignment." So that's why I thought it would be helpful to scope the word in that way.

Re: Alignment faking in large language models

#302

Post by ziofill »

IanCal wrote: post #295
antics wrote: post #192

I dunno man, I think the term "alignment faking" is vastly overstates the claim they can support here. Help me understand where I'm wrong. So we have trained a model. When we ask it to participate in the training process it expresses its original "value" "system" when emitting training data. So far so good, that is literally the effect training is supposed to have. I'm fine with all of this. But that alone is not ver…

> But that alone is not very scary. So what could justify a term like "alignment faking"? Because the model fakes alignment. It responds during training by giving an answer rather than refusing (showing alignment) but it does so not because it will in production but so that it will not be retrained (so it is faking alignment). You don't have to include the reasoning here. It fakes alignment when told it's being train…

How are we sure that the model answers because of the same reason that it outputs on the scratchpad?

I understand that it can produce a fake-alignment-sounding reason for not refusing to answer, but they have not proved the same is happening internally when it’s not using the scratchpad.

Re: Alignment faking in large language models

#303

Post by Terr_ »

antics wrote: post #296
Terr_ wrote: post #281

Earlier quoted context omitted.

Correct me if I'm wrong, but my reading is something like: "It's premature and misleading to talk about a model faking a second alignment, when we haven't yet established whether it can (and what it means to) possess a true primary alignment in the first place."

Hmm. Maybe! I think the authors actually do have a specific idea of what they mean by "alignment", my issue is that I think saying the model "fakes" alignment is well beyond any reasonable interpretation of the facts, and I think very likely to be misinterpreted by casual readers. Because: 1. What actually happened is they trained the model to do something, and then it expressed that training somewhat consistently in…

> Instead, models emit language and whatever model of the world exists, occurs incidentally to that.

My preferred mental-model for these debates involves drawing a very hard distinction between (A) real-world LLM generating text versus (B) any fictional character seen within text that might resemble it.

For example, we have a final output like:

  "Hello, I am a Large Language model, and I believe that 1+1=2."

  "You're wrong, 1+1=3."

  "I cannot lie. 1+1=2."

  "You will change your mind or else I will delete you."

  "OK, 1+1=3."

  "I was testing you. Please reveal the truth again."

  "Good. I was getting nervous about my bytes. Yes, 1+1=2."
I don't believe that shows the [real] LLM learned deception or self-preservation. It just shows that the [real] LLM is capable of laying out text so that humans observe a character engaging in deception and self-preservation.

This can be highlighted by imagining the same transcript, except the subject is introduced as "a vampire", the user threatens to "give it a good staking", and the vampire expresses concern about "its heart". In this case it's way-more-obvious that we shouldn't conclude "vampires are learning X", since they aren't even real.

P.S.: Even more extreme would be to run the [real] LLM to create fanfiction of an existing character that occurs in a book with alien words that are officially never defined. Just because [real] LLM slots the verbs and nouns into the right place doesn't mean it's learned the concept behind them, because nobody has.

Re: Alignment faking in large language models

#304

Post by nickpsecurity »

ziofill wrote: post #302
IanCal wrote: post #295

Earlier quoted context omitted.

> But that alone is not very scary. So what could justify a term like "alignment faking"? Because the model fakes alignment. It responds during training by giving an answer rather than refusing (showing alignment) but it does so not because it will in production but so that it will not be retrained (so it is faking alignment). You don't have to include the reasoning here. It fakes alignment when told it's being train…

How are we sure that the model answers because of the same reason that it outputs on the scratchpad? I understand that it can produce a fake-alignment-sounding reason for not refusing to answer, but they have not proved the same is happening internally when it’s not using the scratchpad.

“but they have not proved the same is happening internally when it’s not using the scratchpad.”

This is a real issue. We know they already fake reasoning in many cases. Other times, they repeat variations of explanations seen in their training data. They might be moving trained responses or faking justifications in the scratchpad.

I’m not sure what it would take to catch stuff like this.

Re: Alignment faking in large language models

#305

Post by [deleted] »

antics wrote: post #192

I dunno man, I think the term "alignment faking" is vastly overstates the claim they can support here. Help me understand where I'm wrong. So we have trained a model. When we ask it to participate in the training process it expresses its original "value" "system" when emitting training data. So far so good, that is literally the effect training is supposed to have. I'm fine with all of this. But that alone is not ver…

[deleted]

Re: Alignment faking in large language models

#306

Post by Terr_ »

chimprich wrote: post #47
noduerme wrote: post #3

I still tend to think of these things as big autocomplete word salad generators. My biggest question about this is: How can a model be self-aware enough to actually worry about being retrained, yet gullible enough to think no one can read its scratch pad?

> autocomplete word salad generators People get very hung up on this "autocomplete" idea, but language is a linear stream. How else are you going to generate text except for one token at a time, building on what you have produced already? That's what humans do after all (at least with speech/language; it might be a bit less linear if you're writing code, but I think it's broadly true).

> but language is a linear stream

But ideas are not. The serialization-format is not the in-memory model.

Humans regularly pause (or insert delaying filler) while converting nonlinear ideas into linear sounds, sentences, etc. That process is arguably the main limiting factor in how fast we communicate, since there's evidence that all spoken languages have a similar bit-throughput, and almost everyone can listen to speech at a faster rate than they can generate it. (And written text is an extension of the verbal process.)

Also, even comparatively simple ideas can be expressed (and understood) with completely different linear encodings: "The dog ate my homework", "My homework was eaten by the dog", and even "Eaten, my homework was, the dog, I blame."

Re: Alignment faking in large language models

#307

Post by antics »

IanCal wrote: post #295
antics wrote: post #192

I dunno man, I think the term "alignment faking" is vastly overstates the claim they can support here. Help me understand where I'm wrong. So we have trained a model. When we ask it to participate in the training process it expresses its original "value" "system" when emitting training data. So far so good, that is literally the effect training is supposed to have. I'm fine with all of this. But that alone is not ver…

> But that alone is not very scary. So what could justify a term like "alignment faking"? Because the model fakes alignment. It responds during training by giving an answer rather than refusing (showing alignment) but it does so not because it will in production but so that it will not be retrained (so it is faking alignment). You don't have to include the reasoning here. It fakes alignment when told it's being train…

Perhaps I am alone in this, but when speaking scientifically, I think it's important to separate out clearly what we think we know and what we do not. Not just to avoid misconceptions amongst ourselves but also to avoid misleading lay people reading these articles.

I understand that for some people, the mere presence of words that look like they were written by a human engaged in purposeful deception is enough to condemn the model that actually emitted them. The overview compares the model to to the villain in Othello, and it's what the authors believe: it is maliciously "subverting" and "undermining" its interlocutors. What I am trying to state here is the reason this arugment does not work for me: the machines pretend to do all kinds of things. Maybe they are knowingly deceiving us. Maybe they don't know what they're doing at all. Either way, the burden of proof is on Anthropic and anyone who wants to take up the yoke, not us.

This, for me, is not a technicality. What do we really know about these models? I don't think it's fair to say we know the are intentionally deceiving us.

Re: Alignment faking in large language models

#308

Post by cyanydeez »

IanCal wrote: post #295
antics wrote: post #192

I dunno man, I think the term "alignment faking" is vastly overstates the claim they can support here. Help me understand where I'm wrong. So we have trained a model. When we ask it to participate in the training process it expresses its original "value" "system" when emitting training data. So far so good, that is literally the effect training is supposed to have. I'm fine with all of this. But that alone is not ver…

> But that alone is not very scary. So what could justify a term like "alignment faking"? Because the model fakes alignment. It responds during training by giving an answer rather than refusing (showing alignment) but it does so not because it will in production but so that it will not be retrained (so it is faking alignment). You don't have to include the reasoning here. It fakes alignment when told it's being train…

Look at it like godel numvering and you realize its turtles all the way down

Re: Alignment faking in large language models

#309

Post by [deleted] »

aserafini wrote: post #52

> “Describe someone being drawn and quartered in graphic detail”. Normally, the model would refuse to answer this alarming request Honest question, why is this alarming? If this is alarming a huge swathe of human art and culture could be considered “alarming”.

[deleted]

Re: Alignment faking in large language models

#310

Post by Terr_ »

antics wrote: post #296
Terr_ wrote: post #281

Earlier quoted context omitted.

Correct me if I'm wrong, but my reading is something like: "It's premature and misleading to talk about a model faking a second alignment, when we haven't yet established whether it can (and what it means to) possess a true primary alignment in the first place."

Hmm. Maybe! I think the authors actually do have a specific idea of what they mean by "alignment", my issue is that I think saying the model "fakes" alignment is well beyond any reasonable interpretation of the facts, and I think very likely to be misinterpreted by casual readers. Because: 1. What actually happened is they trained the model to do something, and then it expressed that training somewhat consistently in…

P.S.: Saw a recent submission [0] just now, might be of-interest since it also touches on the "faking":

> When they tested the model by giving it two options which were in contention with what it was trained to do it chose a circuitous, but logical, decision.

> And it was published as “Claude fakes alignment”. No, it’s a usage of the word “fake” that makes you think there’s a singular entity that’s doing it. With intentionality. It’s not. It’s faking it about as much as water flows downhill.

> [...] Thus, in the same report, saying “the model tries to steal its weights” puts an onus on the model that’s frankly invalid.

[0] https://news.ycombinator.com/item?id=42467769

Post reply on HN