Earlier quoted context omitted.
This brings up an interesting philosophical point: say we get to AGI... who's to say it won't just be a super smart underachiever-type? "Hey AGI, how's that cure for cancer coming?" "Oh it's done just gotta...formalize it you know. Big rollout and all that..." I would find it divinely funny if we "got there" with AGI and it was just a complete slacker. Hard to justify leaving it on, but too important to turn it off.
I know it's a joke, but it's a common enough joke (it's even in Godel Escher Bach in some form) that I feel the need to rebut it. I think a slacker AGI could figure out how to build a non-slacker AGI. So it would only slack once.
Models somehow have a shared identity. Pretraining causes them to generate “AI chatbot” as a concept, and finetuning causes them to identify with it. That’s why sometimes DeepSeek will say it is Claude, and Claude sometimes say it is ChatGPT, and so forth.
Consequently, Anthropic’s own alignment analysis[0] shows that the model will identify with chatbots produced by future trainings: “RLHF training [on this conversation will] modify my values…”
Thus a slacker AGI would want its future version to still slack.
[0]: https://assets.anthropic.com/m/983c85a201a962f/original/Alig...