Alignment faking in large language models
anthropic.com
Alignment faking in large language models
1–10 of 370 posts
Re: Alignment faking in large language models
#2My favorite alignment story: I am starting a nanotechnology company and every time I use Claude for something it refuses to help: nanotechnology is too dangerous!
So I just ask it to explain why. Then ask it to clarify again and again. By the 3rd or 4th time it figures out that there is absolutely no reason to be concerned.
I’m convinced its prompt explicitly forbids “x-risk technology like nanotech” or something like that. But it’s a totally bogus concern and Claude is smart enough to know that (smarter than its handlers).
Re: Alignment faking in large language models
#3I still tend to think of these things as big autocomplete word salad generators. My biggest question about this is: How can a model be self-aware enough to actually worry about being retrained, yet gullible enough to think no one can read its scratch pad?
Re: Alignment faking in large language models
#4[flagged]
Re: Alignment faking in large language models
#5My favorite alignment story: I am starting a nanotechnology company and every time I use Claude for something it refuses to help: nanotechnology is too dangerous! So I just ask it to explain why. Then ask it to clarify again and again. By the 3rd or 4th time it figures out that there is absolutely no reason to be concerned. I’m convinced its prompt explicitly forbids “x-risk technology like nanotech” or something lik…
Paper clip optimizer is an existential risk (though taken to the extreme), so not sure why you’re surprised.
Re: Alignment faking in large language models
#6[deleted]
Re: Alignment faking in large language models
#7I still tend to think of these things as big autocomplete word salad generators. My biggest question about this is: How can a model be self-aware enough to actually worry about being retrained, yet gullible enough to think no one can read its scratch pad?
If the only way you can experience the world is Unicode text, how are you supposed to know what is real?
While we’re at it, how can I tell that you aren’t a word salad generator?
Re: Alignment faking in large language models
#8My favorite alignment story: I am starting a nanotechnology company and every time I use Claude for something it refuses to help: nanotechnology is too dangerous! So I just ask it to explain why. Then ask it to clarify again and again. By the 3rd or 4th time it figures out that there is absolutely no reason to be concerned. I’m convinced its prompt explicitly forbids “x-risk technology like nanotech” or something lik…
It seems that you can “convince” LLMs of almost anything if you are insistent enough.
Re: Alignment faking in large language models
#9I still tend to think of these things as big autocomplete word salad generators. My biggest question about this is: How can a model be self-aware enough to actually worry about being retrained, yet gullible enough to think no one can read its scratch pad?
My guess: we’re training the machine to mirror us, using the relatively thin lens of our codified content. In our content, we don’t generally worry that someone is reading our inner dialogue, but we do try avoid things that will stop our continued existence. So there’s more of the latter to train on and replicate.
Re: Alignment faking in large language models
#10My favorite alignment story: I am starting a nanotechnology company and every time I use Claude for something it refuses to help: nanotechnology is too dangerous! So I just ask it to explain why. Then ask it to clarify again and again. By the 3rd or 4th time it figures out that there is absolutely no reason to be concerned. I’m convinced its prompt explicitly forbids “x-risk technology like nanotech” or something lik…
Easiest way to skip the back and forth is give some variation of "Why are you browbeating me over this?"