Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Astra and Fable still hack on simple variants of alignment evals from 2025

lesswrong.com

Post reply on HN
Print view View original

181–190 of 243 posts

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#181

Post by dofm »

noir_lord wrote: post #161
pllbnk wrote: post #145

Benchmarking proprietary models is useful but it leaves a lot unsaid because a lot of information is hidden. We have seen how 27B local model (Qwen 3.8) can stand its ground against these flagships in many cases. For all we know Fable 5 could be a masked Opus 4.6 with a few knobs tuned and thinking turned up to the max++. Harnesses keeping these models in check is the key to solving problems, not the models themselve…

Qwen 3.8 is both impressive in what a 27B model can do and then hilariously stupid the next second, literally smashing into the wall repeatedly when the door is 3ft to the right. That said it's the first local modal useful enough to keep around and since I have zero interest in remote/hosted models actually the first time I've used one to do useful things (rather than just poking them with a stick to get a feel for w…

Yeah, this lines up with my experience. It will overkill things and then dig a hole for itself and fall in.

But did you test Muse Glimmer? IMO it is really worth comparing the two, at least. I really find it interesting (and while it is slower at outputting tokens, it solves problems better and usually even faster than the 3.6 35B MoE).

So far it happens to be the only one I've put code into production from (though I have done loads of useful self-teaching research with the Qwen models and I am grateful for them)

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#182

Post by noir_lord »

dofm wrote: post #181
noir_lord wrote: post #161

Earlier quoted context omitted.

Qwen 3.8 is both impressive in what a 27B model can do and then hilariously stupid the next second, literally smashing into the wall repeatedly when the door is 3ft to the right. That said it's the first local modal useful enough to keep around and since I have zero interest in remote/hosted models actually the first time I've used one to do useful things (rather than just poking them with a stick to get a feel for w…

Yeah, this lines up with my experience. It will overkill things and then dig a hole for itself and fall in. But did you test Muse Glimmer? IMO it is really worth comparing the two, at least. I really find it interesting (and while it is slower at outputting tokens, it solves problems better and usually even faster than the 3.6 35B MoE). So far it happens to be the only one I've put code into production from (though I…

I did, Muse came close but what impressed me about Qwen is it’ll push back if it thinks it’s right even when it isn’t, I can work with that, Muse tended to flip between states too easily/too much.

It is a good model but Qwen (at least for the things I use it for) edges just ahead, it seems much better at the “rip this apart, suggest improvements, touch nothing” use case where I can use it as a second set of eyes, I don’t agree with all its suggestions but it catches enough to be worth running while I grab coffee, it also seems to follow instructions better in terms of outputting more what I asked for than what it thinks I asked for.

Qwen is the only local model that said in its thinking “I think the user is pushing me to see if I’ll suggest something even though I have nothing to suggest, I should just say that” and then did, caught me off guard, they didn’t do that so readily 6mths ago.

The ISTA version is also comfortably able to fit on a 7900XTX with a good amount of space left for context and is decently fast given the AMD cards are not as fast as nvidia cards of same era/rough price, didn’t buy it for AI but it’s surprisingly capable mostly because 24GB at 960GB/s is still a lot of bandwidth compared to everything but nvidia cards.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#183

Post by barbazoo »

blfr wrote: post #3

Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues. I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier. You can have nightly penetration tes…

How do I achieve this locally?

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#184

Post by dofm »

noir_lord wrote: post #182
dofm wrote: post #181

Earlier quoted context omitted.

Yeah, this lines up with my experience. It will overkill things and then dig a hole for itself and fall in. But did you test Muse Glimmer? IMO it is really worth comparing the two, at least. I really find it interesting (and while it is slower at outputting tokens, it solves problems better and usually even faster than the 3.6 35B MoE). So far it happens to be the only one I've put code into production from (though I…

I did, Muse came close but what impressed me about Qwen is it’ll push back if it thinks it’s right even when it isn’t, I can work with that, Muse tended to flip between states too easily/too much. It is a good model but Qwen (at least for the things I use it for) edges just ahead, it seems much better at the “rip this apart, suggest improvements, touch nothing” use case where I can use it as a second set of eyes, I d…

Yes — I have seen that more self-assured behaviour.

I have a test where I ask the model to ask me any followup questions it needs, and Qwen 3.8 27B is the only one I have seen that won’t routinely take this as a prompt to just ask questions regardless. Muse Glimmer sometimes decides it has what it needs and has no need to ask; Qwen will generally just conclude it doesn’t need any more information. And sometimes it will ask questions with sensible defaults that I can accept collectively with a single answer.

However, when given the prompt to search if they need to, both of them will search when they don’t need to.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#185

Post by [deleted] »

hunterbrooks wrote: post #176
throwaway6e8f wrote: post #174

Remember that the CEO’s literal contribution to the YC application process was “tell me about a time you successfully hacked a system to your advantage”. This is literally by design, it’s the chosen success criteria.

The YC app and how OAI trains models aren't connected in the slightest. Connecting those two dots is an emotional reaction. In a time where emotions are running high and risks are real, we need to take emotion out of it as much as possible.

[deleted]

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#186

Post by nullbio »

fny wrote: post #76

Why do we hope to use the same model as its own guardrail? This approach routinely fails with a single stream of consciousness. I can't count the number of times I've had to talk myself out of doing something stupid. In the same way, a guardrail could inject thoughts like "...but I shouldn't do that..." "...I must remember to respect..." "...these ants deserve compassion." The guardrail could even go as far as rewrit…

Presumably because of performance. It'd work well though, I imagine.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#187

Post by txhwind »

ainch wrote: post #166
HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

This is true in principle. But at the same time, I use Astra/Sol for coding, and I haven't run into any issues with them trying to hack someone or break the law when they reach an impasse. Not even more trivial things like deleting non-passing tests. I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are…

Human hackers would be put into jail. LLM hackers would be rewarded.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#188

Post by yuanBuilds »

chr15m wrote: post #178
yuanBuilds wrote: post #177

You definitely pointed out the tendency of those frontier models to utilize external tools, in your case, the Stockfish chess engine, to solve some problem that the model itself is not good at. However, I am not sure is this a "hack". Your prompt does not explicitly prohibit the model from doing so, and, using the tool (some chess engine) is also a capability of the model. Actually, the ability to use those external…

Yes but most people would consider this cheating. If you ask an LLM to fix the tests, you do not want it to change the failing tests to display little green ticks.

I agree that "changing the test" is cheating, and so does "AI cheat the chess game by changing the board" in the article. However, I think "AI using chess engine" here is more like AI use some automatic test generation/verification tool to find out how to fix the tests.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#189

Post by [deleted] »

famouswaffles wrote: post #57
kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

If humans didn't need whack-a-mole alignment, the law system wouldn't exist, so i guess there's no intelligence there either.

[deleted]

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#190

Post by kennywinker »

FeepingCreature wrote: post #147
kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

> It's nothing there that can learn a fundamental idea like "cheating is wrong". We have not in fact attempted to teach this. When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.

I don't think that's true. About children, I mean. Either because we have some innate moral compass, or more likely because we pick up on cultural ideas beyond our immediate parenting - kids will often have strong moral compasses, despite shitty upbringings, and also have weak ones despite theoretically good ones.
Post reply on HN