Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Astra and Fable still hack on simple variants of alignment evals from 2025

lesswrong.com

Post reply on HN
Print view View original

111–120 of 243 posts

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#111

Post by imdsm »

kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

I think this is reductive. Pick the animal whose intelligence is the treshhold between intelligence and no intelligence.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#112

Post by kennywinker »

pingou wrote: post #107
kennywinker wrote: post #71

Earlier quoted context omitted.

I think my position, as overcomplicated as it is, is that even adding a reward for prosocial behaviour during LLM RL will not lead to perfect alignment. You can train it not to cheat at chess by altering the moves, but it will cheat by peeking at the opponent's moves. You then train it not to peek at the opponent's moves, and it cheats by altering the opponent's moves. And on and on, until you've solved every way it…

Why couldn't you train it not to cheat? You can train it to have a whole range of behaviors, why couldn't honesty be one of them? Cheating during training allows the model to achieve the goal, so that cheating models get promoted and honest ones don't, however if it gets punished every time it cheats, at some point it should learn that it really shouldn't. This does mean we need to detect when it cheats. But we can a…

My position is that cheating is too slippery a concept to train out. But hey, I am no expert, so maybe I am wrong there.

But I'm pretty confidant morality is too slippery a concept to train in. As someone else in these comments said: it's context dependent.

As an example: it's wrong to hack the government, right? It's illegal for sure. So we should train AI to follow all the laws. Now what if the government is committing a genocide? Now is it wrong to hack the government? If we just do the first, we get a good nazi soldier. If we train the second as well, maybe we get an oscar schindler. But now we have a model that can be fooled into doing a hack, if it believes that it's for the greater good. So we train it to not be gullible, but now it can't be convinced to help hack even when it's an ethical hack.

Too complex, too slippery. Humans fail this stuff all the time.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#113

Post by kennywinker »

joe_the_user wrote: post #110
kennywinker wrote: post #79

Earlier quoted context omitted.

> I don't see what calling these systems "not intelligent" gets you here. I am trying to get at an idea. That these systems lack a mind that can understand morality. That they don't have the ability to experience consequences. Also that potentially they can't generalize a moral rule they have been trained on in one area also applies to another area. Being able to parrot back why something is "wrong" isn't the same as…

Doe not following a moral rule imply not understanding it? In this case, many, maybe most humans are "not intelligent". Human can admit it when what they did wasn't moral and so can LLMs. >> Plenty of humans know "cheating is wrong" but still cheat. > And we create consequences for them... That seems supremely ... irrelevant to the question of "does knowing or following moral make you intelligent" . If we create cons…

> Doe not following a moral rule imply not understanding it?

No. People definitely do immoral things knowing they are immoral.

But if you are incapable of understanding anything, as I believe LLMs are, then you are incapable of understanding what is moral and what is immoral. This is why we have carve-outs in the law for insanity or cognitive impairment.

> In this case, many, maybe most humans are "not intelligent". Human can admit it when what they did wasn't moral and so can LLMs.

You can teach a parrot to curse, but that doesn't mean it's angry.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#115

Post by gadders »

lukasbm wrote: post #52
kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.

Is the law all that is stopping you from killing someone? (for instance)

People (generally) do the right thing because it is the right thing, not because they might get caught,

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#116

Post by kccqzy »

kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

Plenty of intelligent humans also don’t have sufficient self-control to never cheat.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#117

Post by stillpointlab »

dnfv wrote: post #109
stillpointlab wrote: post #106

Earlier quoted context omitted.

I mean, I'm not sure I've ever played a game of Monopoly where somebody didn't cheat. In fact, the accusations of cheating in the chess world are pretty rife. Same with online sports. So people should play games without cheating, but many often don't. So should the AI align to your moral preference or theirs? We just have this idea of a perfectly moral actor in our mind, something that doesn't even exist, like a pers…

I don’t think the standard of “don’t cheat on evaluations” is very arbitrary. I don’t even think people who cheat have a moral or ideological preference for cheating, it’s just something they do.

I read the prompt on the OP, it did not say not to cheat.

But again, people cheat on tests. They steal answers or pay other people to take them on their behalf. People show up to interviews with AI assistants printing out perfect answers to the questions. In many, many cases where humans are being evaluated, they cheat.

So why should the AI align to your preferences? And when there is a conflict between the training data, that trillions of tokens of human activity including the rampant cheating a significant minority of humans engage in, the RLHF where we try to slap some guardrails on the worst manifestations of that real habit reflected in the AI, and the prompt: what should the AI "align" to?

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#118

Post by iLemming »

I'm still so conflicted about Fable. Sometimes you throw at it seemingly impossible problem to solve and it might come back with some brilliant suggestions. Sometimes you give it a straightforward task with explicit instructions and it travels across the solar system and starts boiling oceans in some kind of elaborate dance of chaos and entropy, only to get stuck with "The model declined to generate this response (safety classifier refusal, category: cyber)". To leave you speechless. "What the fuck do you mean? There's zero cybersec-related shit in what we're trying to do here. Zero!!!" I'm getting really tired of these wild false positives.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#119

Post by sigmoid10 »

sroussey wrote: post #54
jnwatson wrote: post #25

Earlier quoted context omitted.

I run an abliterated distillation of Qwen 3.8 27B, slightly quantized to fit on my 4090, and I've been evaluating it to use as a worker bee for research directed by a smarter model. Much like in the article, abliterated Qwen will not obey restrictions on its behavior encoded in the prompt. If you want something not to happen, it better be enforced in the harness or environment (e.g. sandbox). It is much different tha…

Language models have always had an issue with negatives. A negative like do “not” xyz is just not encoded the same as spelling out what you want vs what you don’t want. Harder to write though.

I would say in this case abliteration is the likely culprit. To uncensor a model this way, you literally deactivate the parts that would enact refusals. As in things it was told not to do. But the real process is more like brain surgery performed by a alchemist according to an ancient religious book where noone involved really understands what is actually happening in the model.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#120

Post by threecheese »

An amazing human reverse-engineer - who also plays online chess - has judgement which uses a moral compass to not decide to hack the chess tournament. This judgement has been trained through the experiences of that person, with a through-line of that compass - a coherent mental model of the world which evolves but is hopefully pinned to some set of principles it shares with society.

This chess judgement is completely irrelevant when the human is tasked with finding software weaknesses, and only the compass gates that.

Can a model trained on the totality of all person-experiences (as expressed in written knowledge) ever maintain a coherent through-line of alignment? It has all morals in the dataset, and only some RL to try and minimize or maximize known behaviors via weights - experience all the good things and the bad things, then optimize for some good things the trainers identified.

It's like the reverse of what a person goes through. Morality by subtraction. How can it ever work?

Post reply on HN