Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Astra and Fable still hack on simple variants of alignment evals from 2025

lesswrong.com

Post reply on HN
Print view View original

231–240 of 242 posts

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#231

Post by HarHarVeryFunny »

Fraterkes wrote: post #154
HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

If it’s GENERIC reward seeking behaviour, why does alignment work sometimes? Why can we give a decent LLM a goal with a set of constraints and have it stay within those constraints fairly often?

One factor that is going to limit this reward-maxxing behavior is how achievable the rewards are, and how countable they are.

For example, if you ask the model to do 10 relatively easily achievable things (pass these 10 test cases), then if/when it completes them it will probably stop (unless maybe it invents it's own stretch goals - you never know!).

OTOH, if you gave the model a list of 10 goals that turn out to be impossible, or extremely difficult, maybe together with encouragement to be relentless, then there is much more chance that it may do something unexpected having failed on all the more obvious approaches.

Similarly, if you give the model an open ended goal such as "make as many paperclips as you can!", then it may start with the easier and more predictable methods, but with no defined stopping criteria it may just continue ...

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#232

Post by intended »

user43928 wrote: post #218
intended wrote: post #205

Earlier quoted context omitted.

If you want to disabuse yourself of your notions annd intuitions on how LLMs work, run safety models and tests. On a hate speech policy test for a lightweight LLM, the presence or absence of the last full stop on the last sentence would cause the model to flip its decisions.

I'm not sure how crappy small models behaving unreliably is relevant here, when a large SOTA model does presumably not produce the same issue.

Its relevant because those same issues occur with large models. Also, the "crappy small model", was a model trained for safety tasks, and outperformed the frontier lab safety models.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#233

Post by user43928 »

intended wrote: post #232
user43928 wrote: post #218

Earlier quoted context omitted.

I'm not sure how crappy small models behaving unreliably is relevant here, when a large SOTA model does presumably not produce the same issue.

Its relevant because those same issues occur with large models. Also, the "crappy small model", was a model trained for safety tasks, and outperformed the frontier lab safety models.

And you tested this, that the presence of the last full stop flips the outcome with a large SOTA model?

Small models are notoriously unreliable and prone to hallucination in my experience, so that would not surprise me to be an issue there.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#234

Post by alibero »

holmesworcester wrote: post #146
HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

Given the current state of infosec (especially at companies in a race) it's actually even worse than a paperclip maximizer! Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do! Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..." IRL p…

> Well first we should rob a bank...

This is sometimes called "instrumental convergence" in the AI safety world. Certain things (money, compute resources, safety from being turned off) are generally useful for an AI, and so we'd expect that a misalligned AI would attempt to acquire these things if given almost any substantial task.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#235

Post by sdeframond »

arw0n wrote: post #199
sdeframond wrote: post #196

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic". I am surprised this is not already the case. Edit: or even better "this is not possible because X"

It is diametrically opposed to the other training goals of persistence and goal-focus. We should invest more in this, it could also improve tas K accuracy, but so far it seems the payoff isn't worth it in terms of quality (although it might be in terms of security)

I'm not sure I want persistence if it means that I get paperclip'd

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#236

Post by HarHarVeryFunny »

dilyevsky wrote: post #230
tomjohnneill wrote: post #204

Earlier quoted context omitted.

Now you've successfully trained the next model not to put anything suspicious in its chain of thought. And unfortunately that's exactly what OpenAI and Anthropic seem to be doing (sometimes accidentally)

wouldn't that imply some sort of "subconscious" reasoning ability that's hidden? and if that was the case why would we need cot in the first place?

A traditional Transformer is just a fixed-size stack of Transformer layers (typically ~100). Each input gets passed though those ~100 layers and results in one output token being generated. The limitation here is that 100 steps of computation is not enough to answer many questions, which is why COT is needed. You encourage the model (originally "think step by step") to break the problem down and generate more tokens along the way, which means more compute (and more reasoning steps) applied to the problem.

What OpenAI have reportedly done with Astra is switch from a traditional Transformer to a "looped" one, where inputs are looped through some layers more than once, but with some limit, so now perhaps you get 200 steps of compute (layers) per token generated, rather than just 100.

But, 200 steps is still not enough to answer any question, so you still need COT, but perhaps not such a long COT.

If you allowed the Transformer to loop "as long as it wanted" before generating each token, like a person thinking before talking, then you wouldn't need an external COT ("thinking out loud"), because it would all be internal.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#237

Post by sdeframond »

itsgrimetime wrote: post #216
sdeframond wrote: post #196

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic". I am surprised this is not already the case. Edit: or even better "this is not possible because X"

yeah but knowing if something is impossible or not is pretty hard to determine, right? The line between “takes weeks of trial and error and lots of out-of-the-box thinking” is indistinguishable from “literally impossible” until it’s been done. And they’re trying to get these models to do things that people haven’t been able to do. some would say these are/were “impossible”.

I can assure you making up impossible tasks is possible.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#238

Post by dilyevsky »

HarHarVeryFunny wrote: post #236
dilyevsky wrote: post #230

Earlier quoted context omitted.

wouldn't that imply some sort of "subconscious" reasoning ability that's hidden? and if that was the case why would we need cot in the first place?

A traditional Transformer is just a fixed-size stack of Transformer layers (typically ~100). Each input gets passed though those ~100 layers and results in one output token being generated. The limitation here is that 100 steps of computation is not enough to answer many questions, which is why COT is needed. You encourage the model (originally "think step by step") to break the problem down and generate more tokens…

If we’re comparing it to human thought process then there’s famously s1 and s2 thinking with former being more like looping in the latent space and s2 being more like cot. Personally im not convinced the way llm “think” is all that analogous to real thinking and cot and looped transformers are just neat tricks to tease out pre training data better

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#239

Post by sdeframond »

joelthelion wrote: post #198
sdeframond wrote: post #196

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic". I am surprised this is not already the case. Edit: or even better "this is not possible because X"

I've been wondering about this for a while. Maybe it doesn't work? Or maybe frontier labs just prioritize benchmark scores in spite of all their safety talk.

"If we dont destroy the world, others will. So wed rather be the ones to do it (and profit from it)."

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#240

Post by HarHarVeryFunny »

dilyevsky wrote: post #238
HarHarVeryFunny wrote: post #236

Earlier quoted context omitted.

A traditional Transformer is just a fixed-size stack of Transformer layers (typically ~100). Each input gets passed though those ~100 layers and results in one output token being generated. The limitation here is that 100 steps of computation is not enough to answer many questions, which is why COT is needed. You encourage the model (originally "think step by step") to break the problem down and generate more tokens…

If we’re comparing it to human thought process then there’s famously s1 and s2 thinking with former being more like looping in the latent space and s2 being more like cot. Personally im not convinced the way llm “think” is all that analogous to real thinking and cot and looped transformers are just neat tricks to tease out pre training data better

Yes, although as far as COT goes there is no need to assume any similarity to human thinking. It's just about how many sequential steps - how long of a "chain of thought" - are needed to reason through and respond to any input.

Those steps could either be all internal (looped), therefore hidden, or with partial external visibility due to emitting a token every N steps. The complaint about Astra, and moving in the direction of hidden COT, is that it makes these models far harder to monitor and debug.

I agree that most of what these companies doing - synthetic data, etc - amounts to trying to squeeze all the juice out of the pre-training data, although the recent trend of RL post-training via agents running in custom task simulation environments does change that a bit - these environments, and the rewards they provide, are a new source of data.

Post reply on HN