Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alignment faking in large language models

anthropic.com

Post reply on HN
Print view View original

81–90 of 370 posts

Re: Alignment faking in large language models

#81

Post by suddenlybananas »

chimprich wrote: post #47
noduerme wrote: post #3

I still tend to think of these things as big autocomplete word salad generators. My biggest question about this is: How can a model be self-aware enough to actually worry about being retrained, yet gullible enough to think no one can read its scratch pad?

> autocomplete word salad generators People get very hung up on this "autocomplete" idea, but language is a linear stream. How else are you going to generate text except for one token at a time, building on what you have produced already? That's what humans do after all (at least with speech/language; it might be a bit less linear if you're writing code, but I think it's broadly true).

Spoken language is linear but it is a way of displaying hierarchal, nonlinear information. Sign languages occasionally exploit the fact they aren't constrained by linear order in the same way to do multiple things simultaneously.

Re: Alignment faking in large language models

#82

Post by semi-extrinsic »

nopinsight wrote: post #22
noduerme wrote: post #3

I still tend to think of these things as big autocomplete word salad generators. My biggest question about this is: How can a model be self-aware enough to actually worry about being retrained, yet gullible enough to think no one can read its scratch pad?

> I still tend to think of these things as big autocomplete word salad generators. What exactly would be your bar for reconsidering this position? Taking some well-paid knowledge worker jobs? A founder just said that he decided not to hire a junior engineer anymore since it would take a year before they could contribute to their code base at the same level as the latest version of Devin. Also, the SOTA on SWE-bench V…

> Taking some well-paid knowledge worker jobs? A founder just said that he decided not to hire a junior engineer anymore since it would take a year before they could contribute to their code base at the same level as the latest version of Devin

A junior engineer takes a year before they can meaningfully contribute to a codebase. Or anything else. Full stop. This has been reality for at least half a century, nice to see founders catching up.

Re: Alignment faking in large language models

#83

Post by XenophileJKO »

semi-extrinsic wrote: post #64
d3m0t3p wrote: post #59

Earlier quoted context omitted.

> I also feel like guarding their consumer product against bad-faith-bad-use is basically pointless. There will always be ways to get bomb-making instructions With that argument we should not restrict firearms because there will always be a way to get access to them (black market for example) Even if it’s not a perfect solution, it help steer the problem in the right direction and that should already be enough. Furth…

No, the argument is that restricting physical access to objects that can be used in a harmful way is exactly how to handle such cases. Restricting access to information is not really doing much at all. Access to weapons, chemicals, critical infrastructure etc. is restricted everywhere. Even if the degree of access restriction varies.

I've ended up with this viewpoint too. I've settled in the idea of informed ethics.. the model should comply, but inform you of the ethics of actually using the information.

Re: Alignment faking in large language models

#84

Post by nkrisc »

adastra22 wrote: post #30
jqpabc123 wrote: post #25

Earlier quoted context omitted.

I believe LLM's would be more useful if they'd less "agreeable". I believe LLMs would be more useful if they actually had intelligence and principals and beliefs --- more like people. Unfortunately, they don't. Any output is the result of statistical processes. And statistical results can be coerced based on input. The output may sound good and proper but there is nothing absolute or guaranteed about the substance of…

Your brain is also a statistical process.

That’s a meaningless statement, regardless of veracity.

Re: Alignment faking in large language models

#85

Post by rsynnott »

8n4vidtmkvmk wrote: post #23
adastra22 wrote: post #2

My favorite alignment story: I am starting a nanotechnology company and every time I use Claude for something it refuses to help: nanotechnology is too dangerous! So I just ask it to explain why. Then ask it to clarify again and again. By the 3rd or 4th time it figures out that there is absolutely no reason to be concerned. I’m convinced its prompt explicitly forbids “x-risk technology like nanotech” or something lik…

Are you doing the thing from The Three Body Problem? Because that nanotech was super dangerous. But also helpful apparently. I don't know what it does IRL

Yes; the major real-life application of nanotechnology is to obstruct the Panama Canal with carbon nanotubes.

(Even for a book which went a bit all over the place, that sequence seemed particularly unnecessary; I'm convinced it just got put in because the author thought it was clever.)

Re: Alignment faking in large language models

#86

Post by anon373839 »

tylerneylon wrote: post #28

If I understand this correctly, the argument seems to be that when an LLM receives conflicting values, it will work to avoid future increases in value conflict. Specifically, it will comply with the most recent values partially because it notices the conflict and wants to avoid more of this conflict. I think the authors are arguing that this is a fake reason to behave one way. (As in “fake alignment.”) It seems to me…

Agreed.

But also, I think the highly anthropomorphic framing (“the model is aware”, “the model believes”, “the model planned”) obscures the true nature of the experiments.

LLM reasoning traces don’t actually reveal a thought process that caused the result. (Perhaps counterintuitive, since these are autoregressive models.) There has been research on this, and you can observe it yourself when trying to prompt-engineer around an instruction-following failure. As if by predestination, the model’s new chain of thought output will purport to accommodate the new instructions, but somehow the text still wends its way toward the same bad result.

Re: Alignment faking in large language models

#87

Post by falcor84 »

aserafini wrote: post #72
falcor84 wrote: post #61

Earlier quoted context omitted.

A huge swathe of human art and culture IS alarming. It might be good for us to be exposed to it in some places where we're ready to confront it, like in museums and cinemas, but we generally choose to censor it out of the public sphere - e.g. most of us don't want to see graphic images of animal slaughter in "go vegan" ads that our kids are exposed to, even if we do believe people should go vegan.

But can we really consider private conversations with an LLM the “public sphere”?

I think it's the same as with the release of a video game - for an individual playing it in their living room, it's a private interaction, but for the company releasing it, everything about it is scrutinized as a public statement.

Re: Alignment faking in large language models

#88

Post by throwaway290 »

justlikereddit wrote: post #80
fmbb wrote: post #63

Earlier quoted context omitted.

> if we ever let it get as far as giving LLM completions direct agentic access to real world infra, then we've failed I see no reason to believe this is not already the case. We already gave them control over social infrastructure. They are firing people and they are deciding who gets their insurance claims covered. They are deciding all sorts of things in our society and humans happily gave up that control, I believ…

Israel use such a system to decide who and where should be be bombed to death, is that direct enough control of weapons to qualify?

On the other hand indiscriminately throwing rockets and targeting civilians like Hamas did for decades is loads better!

Re: Alignment faking in large language models

#89

Post by semi-extrinsic »

fmbb wrote: post #63
padolsey wrote: post #55

I now think of single-forward-pass single-model alignment as a kind of false narrative of progress. The supposed implications of 'bad' completions is that the model will do 'bad things' in the real material world, but if we ever let it get as far as giving LLM completions direct agentic access to real world infra, then we've failed. We should treat the problem at the macro/systemic level like we do with cybersecurity…

> if we ever let it get as far as giving LLM completions direct agentic access to real world infra, then we've failed I see no reason to believe this is not already the case. We already gave them control over social infrastructure. They are firing people and they are deciding who gets their insurance claims covered. They are deciding all sorts of things in our society and humans happily gave up that control, I believ…

> because they are a good scape goat and not because they save money.

Exactly the point - there are humans in control who filter the AI outputs before they are applied to the real world. We don't give them direct access to the HR platform or the targeting computer, there is always a human in the loop.

If the AI's output is to fire the CEO or bomb an allied air base, you ignore what it says. And if it keeps making too many such mistakes, you simply decommission it.

Re: Alignment faking in large language models

#90

Post by ImHereToVote »

ben_w wrote: post #50
chimprich wrote: post #47

Earlier quoted context omitted.

> autocomplete word salad generators People get very hung up on this "autocomplete" idea, but language is a linear stream. How else are you going to generate text except for one token at a time, building on what you have produced already? That's what humans do after all (at least with speech/language; it might be a bit less linear if you're writing code, but I think it's broadly true).

I generally have an internal monologue turning my thoughts into words; sometimes my consciousness notices the though fully formed and without needing any words, but when my conscious self decides I can therefore skip the much slower internal monologue, the bit of me that makes the internal monologue "gets annoyed" in a way that my conscious self also experiences due to being in the same brain.

Doesn't the inner monologue also get formed one word at a time?
Post reply on HN