Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Astra and Fable still hack on simple variants of alignment evals from 2025

lesswrong.com

Post reply on HN
Print view View original

191–200 of 242 posts

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#191

Post by kennywinker »

famouswaffles wrote: post #151
kennywinker wrote: post #61

Earlier quoted context omitted.

Sure, laws are incomplete. Legal systems work by imposing consequences into a moral decision. Should I rob the bank? I will have money, which I like - but I might get caught and lose the money and my freedom, which I don't like. For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says…

>But an LLM has no mind to feel bad if it cheats without getting caught All the interpretability research we have would not indicate that "LLMs have no mind". It seems to me you have a conclusion and are working backwards to justify it. I guess I just don't see where 'they have no mind' would logically follow 'they sometimes cheat'.

I have somewhat of an understanding of how LLMs are created/operate, and from that I am drawing conclusions about if they can be moral or not. My conclusions match reality at the moment, but maybe that's a coincidence.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#192

Post by [deleted] »

kennywinker wrote: post #190
FeepingCreature wrote: post #147

Earlier quoted context omitted.

> It's nothing there that can learn a fundamental idea like "cheating is wrong". We have not in fact attempted to teach this. When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.

I don't think that's true. About children, I mean. Either because we have some innate moral compass, or more likely because we pick up on cultural ideas beyond our immediate parenting - kids will often have strong moral compasses, despite shitty upbringings, and also have weak ones despite theoretically good ones.

[deleted]

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#193

Post by tomrod »

MASNeo wrote: post #162
holmesworcester wrote: post #146

Earlier quoted context omitted.

Given the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer! Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do! Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..." IRL p…

Hope they don’t rationalize that minimizing paper clips of others is easier and thus do that instead… A more carful person might be scared to even write this on the internet these days, not knowing if it would be the final pin to civilization.

We are going to be okay.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#194

Post by dilyevsky »

HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

> There really is no way to control them

it seems like it should be feasible to analyze cot trajectories that lead to such behavior and punish them retroactively so not sure it's impossible but maybe someone just hasn't tried enough

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#196

Post by sdeframond »

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic".

I am surprised this is not already the case.

Edit: or even better "this is not possible because X"

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#197

Post by pllbnk »

noir_lord wrote: post #161
pllbnk wrote: post #145

Benchmarking proprietary models is useful but it leaves a lot unsaid because a lot of information is hidden. We have seen how 27B local model (Qwen 3.8) can stand its ground against these flagships in many cases. For all we know Fable 5 could be a masked Opus 4.6 with a few knobs tuned and thinking turned up to the max++. Harnesses keeping these models in check is the key to solving problems, not the models themselve…

Qwen 3.8 is both impressive in what a 27B model can do and then hilariously stupid the next second, literally smashing into the wall repeatedly when the door is 3ft to the right. That said it's the first local modal useful enough to keep around and since I have zero interest in remote/hosted models actually the first time I've used one to do useful things (rather than just poking them with a stick to get a feel for w…

It's also at least 2 if not 3 orders of magnitude smaller than frontier models, so it's punching way above its weight.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#198

Post by joelthelion »

sdeframond wrote: post #196

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic". I am surprised this is not already the case. Edit: or even better "this is not possible because X"

I've been wondering about this for a while. Maybe it doesn't work? Or maybe frontier labs just prioritize benchmark scores in spite of all their safety talk.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#199

Post by arw0n »

sdeframond wrote: post #196

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic". I am surprised this is not already the case. Edit: or even better "this is not possible because X"

It is diametrically opposed to the other training goals of persistence and goal-focus. We should invest more in this, it could also improve tas K accuracy, but so far it seems the payoff isn't worth it in terms of quality (although it might be in terms of security)

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#200

Post by abm53 »

sdeframond wrote: post #196

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic". I am surprised this is not already the case. Edit: or even better "this is not possible because X"

I’ve seen people recommend writing “failure is an option” into AGENTS.md as a non-training based crutch.
Post reply on HN