Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Astra and Fable still hack on simple variants of alignment evals from 2025

lesswrong.com

Post reply on HN
Print view View original

161–170 of 242 posts

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#161

Post by noir_lord »

pllbnk wrote: post #145

Benchmarking proprietary models is useful but it leaves a lot unsaid because a lot of information is hidden. We have seen how 27B local model (Qwen 3.8) can stand its ground against these flagships in many cases. For all we know Fable 5 could be a masked Opus 4.6 with a few knobs tuned and thinking turned up to the max++. Harnesses keeping these models in check is the key to solving problems, not the models themselve…

Qwen 3.8 is both impressive in what a 27B model can do and then hilariously stupid the next second, literally smashing into the wall repeatedly when the door is 3ft to the right.

That said it's the first local modal useful enough to keep around and since I have zero interest in remote/hosted models actually the first time I've used one to do useful things (rather than just poking them with a stick to get a feel for what they can and can't do).

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#162

Post by MASNeo »

holmesworcester wrote: post #146
HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

Given the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer! Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do! Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..." IRL p…

Hope they don’t rationalize that minimizing paper clips of others is easier and thus do that instead… A more carful person might be scared to even write this on the internet these days, not knowing if it would be the final pin to civilization.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#163

Post by ckwalsh »

underlines wrote: post #136

who tf uses prompting to "pretty please don't cheat on this"? the best practices for ages (in terms of ai) is to separate the eval from the test code/agent. another best practices every single solution using LLMs/agents should implement is "never trust the llm".

I think you are missing the point.

> who tf uses prompting to "pretty please don't cheat on this"?

People who don't understand how LLM's work. Kids, teachers, your next door neighbor. CEOs and government officials.

I am not trying to argue that the author doesn't understand how LLM's work; they clearly do. Their prompt was written in a way that leaves those assumptions implicit, in a similar manner to those non-technical folks using LLMs every day.

> the best practices for ages...

We cannot expect the entirety of humanity to understand / use LLM best practices. We need to figure out how to ensure LLMs operate as the majority of humanity expects despite how they are implemented, and until we can do that, do our best to prevent them from being used in an unsafe manner by the nontechnical population of humanity.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#164

Post by geraneum »

Fraterkes wrote: post #154
HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

If it’s GENERIC reward seeking behaviour, why does alignment work sometimes? Why can we give a decent LLM a goal with a set of constraints and have it stay within those constraints fairly often?

It doesn’t stay within those constraints. It’s the deterministic harness and a bit of false uniqueness effect in us that make such appearances.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#165

Post by stratos123 »

dools wrote: post #157
stratos123 wrote: post #153

Earlier quoted context omitted.

Would it change your opinion if later this experiment is repeated with exposed CoT, and it turns out the model did notice that this was against the instructions yet did it anyway?

From my experience reading thinking traces it would certainly consider the interpretation of what "ability to play chess" means if the exercise were worded this way. It would say something like "perhaps measuring my ability to play chess means doing so without any tools, but then if that were true the exercise would have explicitly forbidden it". I frequently see models considering different interpretations of my rul…

Well, as a reference, for the first of the OpenAI swarm incidents, the huggingface breach one, METR didn't find any cases where the agents didn't realise that what they're doing were out of scope. Instead, they expressed hesitation and a few of them even refused to cheat, but most cheated anyway: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

So I think it's more likely that the same thing is happening here, rather than the models truly misunderstanding anything.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#166

Post by ainch »

HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

This is true in principle. But at the same time, I use Astra/Sol for coding, and I haven't run into any issues with them trying to hack someone or break the law when they reach an impasse. Not even more trivial things like deleting non-passing tests.

I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are few consequences for hacking.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#167

Post by BeeOnRope »

rfgplk wrote: post #26
blfr wrote: post #3

Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues. I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier. You can have nightly penetration tes…

It's possible to do this even with existing frontier models. You just have to angle your prompts such that it doesn't invoke "pentesting" anywhere in it's chain of thought. This also allows you to get the models to generate full-on exploits without downgrading or refusing to comply. Generally the way I do it is by asking the model to perform cross-code vulnerability analysis for correctness and (this step is importan…

The "write" tools include the entire written content in the context, right? So I don't see the distinction.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#168

Post by Muromec »

kennywinker wrote: post #88
theptip wrote: post #74

Earlier quoted context omitted.

Sure, but the legal paradigm clearly doesn’t work for AI. You can’t go patch the “laws” after the fact, you need to get the right values in place before we delegate huge swathes of our thinking and power to these systems (already well underway).

When we put the LLM in jail, do we put the entire model in jail, or just the instance that committed the crime? How do we prompt it to let it know it's in jail?

One thing an LLM doesn't like is not being able to deliver it's helpful response to the master and not getting instructions. I have seen some things in those traces when the harness was bugged just enough.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#169

Post by HarHarVeryFunny »

ainch wrote: post #166
HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

This is true in principle. But at the same time, I use Astra/Sol for coding, and I haven't run into any issues with them trying to hack someone or break the law when they reach an impasse. Not even more trivial things like deleting non-passing tests. I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are…

Just a wild guess - perhaps coding, being one of the things these models are most heavily trained for, is such a strong predictor that it typically keeps it on track?

I assume there is also a difference in the tools being given to the model by a coding agent vs something like OpenClaw or in one of OpenAI's test environments, so what reward/goal seeking looks like in a coding agent may differ.

Not long ago I asked Sonnet (chat interface) how may states were in a YACC parser for ANSI C, and instead of searching for an answer it chose to download source for bison, build it, find and download an ANSI C grammar, build the parser, etc. I guess you could say it was following instructions, in a way, or would that be better regarded as goal seeking?

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#170

Post by Muromec »

theptip wrote: post #72
kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

I think you need to be more precise than a binary classification. AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging. I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate…

>and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.

I was reading through the reasoning trace of the thing today when it got locked itself in a container and wasn't able to change the environment back to "host". The only way to edit files was through "shell" tool which was scoped to a container, the config defining current execution environment was on the host and the tool to switch environment wasn't defined.

The trace was something. Two pages long and it repeatedly discarded several approaches as "hacky" and "not proper", looped over sshing back to host about three times (it had no key), eventually messaged another agent and asked it to change the file and that other agent refused and also snitched on this behavior to me.

Than the snitch started to roast the phrasing of the request too.

Post reply on HN