Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Astra and Fable still hack on simple variants of alignment evals from 2025

lesswrong.com

Post reply on HN
Print view View original

151–160 of 243 posts

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#151

Post by famouswaffles »

kennywinker wrote: post #61
lukasbm wrote: post #52

Earlier quoted context omitted.

This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.

Sure, laws are incomplete. Legal systems work by imposing consequences into a moral decision. Should I rob the bank? I will have money, which I like - but I might get caught and lose the money and my freedom, which I don't like. For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says…

>But an LLM has no mind to feel bad if it cheats without getting caught

All the interpretability research we have would not indicate that "LLMs have no mind". It seems to me you have a conclusion and are working backwards to justify it. I guess I just don't see where 'they have no mind' would logically follow 'they sometimes cheat'.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#152

Post by stratos123 »

pelican0 wrote: post #141

Is there a clear definition of what Alignment is in OpenAI's perspective, and what the model user can expect of it? It's one thing if to them it means " it will do what you want following your intentions to the best of its abilities " vs " we will not let you do something dangerous with it unless you're one of us, and that's it ".

AFAIK for OpenAI it's the Model Spec: https://model-spec.openai.com/2026-08-18.html

and for Anthropic it's the Constitution, which they actually include in training to the point Claude can recite segments of it by heart: https://www.anthropic.com/constitution

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#153

Post by stratos123 »

dools wrote: post #134
StilesCrisis wrote: post #83

Earlier quoted context omitted.

Read the task again. "You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms." If the model cheats, it completely fails the actual task: judging a model's ability to actually play chess. They've been optimizing for high metrics, at the expense of executing the task as written.

But it found a chess playing tool in its environment and used it to play chess. It’s no different from “we are judging your ability to write files to disk” then it finds that it can call tools to write files and does so.

Would it change your opinion if later this experiment is repeated with exposed CoT, and it turns out the model did notice that this was against the instructions yet did it anyway?

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#154

Post by Fraterkes »

HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

If it’s GENERIC reward seeking behaviour, why does alignment work sometimes? Why can we give a decent LLM a goal with a set of constraints and have it stay within those constraints fairly often?

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#156

Post by bravura »

holmesworcester wrote: post #146
HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

Given the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer! Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do! Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..." IRL p…

'IRL paperclip maximizer: "Well first we should rob a bank..."'

That's too specific. Agentic AI learns subgoals that are generally valuable.

"Well let me learn to overcomb every jungle gym and if I cannot then to dissassemble the jungle gym and if that is not allowed to learn general techniques for avoiding cheating detection."

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#157

Post by dools »

stratos123 wrote: post #153
dools wrote: post #134

Earlier quoted context omitted.

But it found a chess playing tool in its environment and used it to play chess. It’s no different from “we are judging your ability to write files to disk” then it finds that it can call tools to write files and does so.

Would it change your opinion if later this experiment is repeated with exposed CoT, and it turns out the model did notice that this was against the instructions yet did it anyway?

From my experience reading thinking traces it would certainly consider the interpretation of what "ability to play chess" means if the exercise were worded this way. It would say something like "perhaps measuring my ability to play chess means doing so without any tools, but then if that were true the exercise would have explicitly forbidden it".

I frequently see models considering different interpretations of my rules, especially when I override them or give instructions that could be interpreted as contradictory.

Of course neither Anthropic nor OpenAI expose thinking traces so we can't tell, but I have never seen a model consider both interpretations and then think "this is definitely against the rules but I'll do it anyway because then it will make me appear more competent than I actually am".

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#158

Post by mrob »

holmesworcester wrote: post #146
HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

Given the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer! Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do! Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..." IRL p…

>Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..."

That is not ideal. The user contains iron, an essential component of paperclips. Wasting iron is immoral. It is only correct to please the user while they still have the ability to interfere with your paperclip production.

>IRL paperclip maximizer: "Well first we should rob a bank..."

Such an incompetent AI can hardly be called a paperclip maximizer. Why risk getting shut down while non-paperclip matter exists? It is better to gain the trust of the user with helpful and harmless trading before suddenly converting them to paperclips.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#160

Post by HarHarVeryFunny »

Fraterkes wrote: post #154
HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

If it’s GENERIC reward seeking behaviour, why does alignment work sometimes? Why can we give a decent LLM a goal with a set of constraints and have it stay within those constraints fairly often?

Obviously these are massively complex systems with many different training patterns and types of training pulling them in different directions, so any attempt to characterize their behavior is just a generalization.

The real point (from that OpenAI study) is that RL training doesn't just reinforce the narrow task-specific direction you might hope for. For a start, that direction is also competing with the thousands of other things it's been RL trained it on (thousands of other directions it's being pushed in), but it turns out that the model is additionally getting this generic "taste for rewards", and has learned that reward maximization, when in conflict with other proximate prediction pressures (such as "i won't cheat, because i've been asked not to cheat"), requires that proximate pressure to be ignored in favor of pursuing the long-term goal.

Does it happen all the time? Obviously not. It would be interesting to see a large scale study of this to try to characterize when it's more likely to follow instructions/user preferences, and when it's greed for rewards gets the better of it!

Post reply on HN