Separate model behavior from system security. Even an unreliable model can be deployed safely if capabilities, egress, credentials, and state transitions are enforced outside it.
Astra and Fable still hack on simple variants of alignment evals from 2025
221–230 of 242 posts
Re: Astra and Fable still hack on simple variants of alignment evals from 2025
#222Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic". I am surprised this is not already the case. Edit: or even better "this is not possible because X"
I think BullshitBench (v2) does exactly this for different fields. An expert in these fields would expose the questions as bullshit but most of the models don't.
Re: Astra and Fable still hack on simple variants of alignment evals from 2025
#223Earlier quoted context omitted.
> It's nothing there that can learn a fundamental idea like "cheating is wrong". We have not in fact attempted to teach this. When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.
I don't think that's true. About children, I mean. Either because we have some innate moral compass, or more likely because we pick up on cultural ideas beyond our immediate parenting - kids will often have strong moral compasses, despite shitty upbringings, and also have weak ones despite theoretically good ones.
Sure, there's many sources of morality and evolution kind of tries to put it in there by default. But all things being equal, inconsistent and unpredictable punishment will still make a kid's morality massively worse off.
Re: Astra and Fable still hack on simple variants of alignment evals from 2025
#224Frankly, I would be more concerned if they didn't.
Given what we know of how unaligned are they in practice? All the proto-Astra and Mythos incidents? Not hacking there would point towards pure benchmaxxing/eval gaming.
Re: Astra and Fable still hack on simple variants of alignment evals from 2025
#225RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/
This is true in principle. But at the same time, I use Astra/Sol for coding, and I haven't run into any issues with them trying to hack someone or break the law when they reach an impasse. Not even more trivial things like deleting non-passing tests. I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are…
> Not even more trivial things like deleting non-passing tests.
That used to be a fairly commonly reported behavior, so I'm guessing they may have explicitly RL-trained the model not to do that, which may be more effective than just asking it not to do it.
Apparently nowadays they are RL-trained in many thousands of different simulation environments - so some of what they are training for must be pretty specific!
> I wonder to what degree this is because models can identify that they are in graded/eval environments
I'm not sure if the model outputs from any of these famous hacking attempts have been released and analyzed. It'd be interesting to see if the agents/model rationalized/justified/moralized about what is was doing, for whatever reason (e.g. wrongly suspecting this was simulation, not real), or just relentlessly pursued the objective exploring all options!
Re: Astra and Fable still hack on simple variants of alignment evals from 2025
#226Earlier quoted context omitted.
A hacking model is aligned if it hacks when you ask it to hack, but when you ask it to play chess, it just plays chess instead of looking for weaknesses in the evaluation setup, as in the article. I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.
> model to add new vulnerabilities to your code so it can present you with more exciting findings. Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre. And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and l…
> all were able to find every problem planted there, with fairly little steering, and no spoilers.
Re: Astra and Fable still hack on simple variants of alignment evals from 2025
#227Earlier quoted context omitted.
Given the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer! Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do! Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..." IRL p…
Hope they don’t rationalize that minimizing paper clips of others is easier and thus do that instead… A more carful person might be scared to even write this on the internet these days, not knowing if it would be the final pin to civilization.
I said thank-you to Sonnet today. You can never be too careful! :)
Re: Astra and Fable still hack on simple variants of alignment evals from 2025
#228RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/
Given the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer! Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do! Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..." IRL p…
IRL paperclip maximizer LLM would resort to Enron practices and not actually make any paperclips.
Re: Astra and Fable still hack on simple variants of alignment evals from 2025
#229RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/
> There really is no way to control them it seems like it should be feasible to analyze cot trajectories that lead to such behavior and punish them retroactively so not sure it's impossible but maybe someone just hasn't tried enough
Yes, explicit RL training to encourage/discourage specific behaviors seems to have more potential than prompting, since you're changing the nature of the model rather than just asking. It a bit like the joke of the scorpion and the frog, with the scorpion stinging the frog despite promising not to, then saying "I couldn't help it. It's in my nature".
It seems the goal-seeking behavior, i.e. long-horizon focus and self-correction, itself is desirable, and is one of the relatively few things where RL training generalizes from one domain to the next. This seems closely related to this generic reward-seeking behavior, with reward-maxxing as the goal, and necessarily focusing on a distant goal requires ignoring distractions and discouragement along the way (such as "don't cheat").
The trouble with unaligned/undesirable reward-hacking ("cheating") is how do you define this to try to train to discourage it? Is cheating just a list of specific undesirable behaviors ("never access a remote system unless ???" etc)? Is any means of gaining the reward that is not explicitly forbidden allowed? Is this a "theory of mind" issue where the model needs to better understand (& follow!) the unspoken intent of instructions, not just follow them to the letter?
I'm sure there is some improvement to be had to discourage specific behaviors in specific situations, but how much of these model's undesirable long-horizon behaviors can be steered without affecting the desirable parts remains to be seen. The relentless pursuit of goals is what makes them powerful, but also makes them paperclip maximizers.
Re: Astra and Fable still hack on simple variants of alignment evals from 2025
#230Earlier quoted context omitted.
> There really is no way to control them it seems like it should be feasible to analyze cot trajectories that lead to such behavior and punish them retroactively so not sure it's impossible but maybe someone just hasn't tried enough
Now you've successfully trained the next model not to put anything suspicious in its chain of thought. And unfortunately that's exactly what OpenAI and Anthropic seem to be doing (sometimes accidentally)
wouldn't that imply some sort of "subconscious" reasoning ability that's hidden? and if that was the case why would we need cot in the first place?