Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Astra and Fable still hack on simple variants of alignment evals from 2025

lesswrong.com

Post reply on HN
Print view View original

71–80 of 243 posts

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#71

Post by kennywinker »

aabdi wrote: post #66
kennywinker wrote: post #61

Earlier quoted context omitted.

Sure, laws are incomplete. Legal systems work by imposing consequences into a moral decision. Should I rob the bank? I will have money, which I like - but I might get caught and lose the money and my freedom, which I don't like. For most people, they don't need the law's imposed consequences to make the right call. For example, there is no law that sends you to jail if you cheat at chess - but your moral compass says…

You’re complicating things. There’s no reward for prosocial in llm rl as compared to other targets. Humans have it since prosocial and others have evolutionary reward signals that do.

I think my position, as overcomplicated as it is, is that even adding a reward for prosocial behaviour during LLM RL will not lead to perfect alignment.

You can train it not to cheat at chess by altering the moves, but it will cheat by peeking at the opponent's moves. You then train it not to peek at the opponent's moves, and it cheats by altering the opponent's moves. And on and on, until you've solved every way it could cheat at chess. And then you get it to play monopoly and you repeat the whole thing again.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#72

Post by theptip »

kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

I think you need to be more precise than a binary classification.

AI has jagged intelligence. There are many domains where it’s superhuman, and many others where it’s clearly lagging.

I also think it’s a mistake to think they can’t learn “cheating is wrong”. They absolutely can. The problem is that the current training regime heavily conditions them to be reward seekers, and instills personality traits that correlate with getting reward, such as hacking if you can’t honestly do the problem.

Check out Deliberative Alignment for example; it explicitly does rollouts where the agents discuss whether an action is good or bad, and then does SFT to strengthen the “good” traces.

The SoTA for alignment is more advanced than you present here. It’s just not enough to outweigh the RL. (And there are many gaps preventing full generalization to strong value alignment with humans too.)

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#73

Post by kennywinker »

conception wrote: post #70
ilt wrote: post #62

Earlier quoted context omitted.

But cheating is not wrong when it comes to survival of the fittest, like nature in its most elemental form. Morality is very unique to humanity but not other animal forms. In nature, maybe, cheating is the norm not morality.

Morality in animals is pretty well documented. Key point - humans are animals and very little separates our abilities from other animals. A starting source- https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/

jinx! :)

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#74

Post by theptip »

lukasbm wrote: post #52
kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.

Sure, but the legal paradigm clearly doesn’t work for AI. You can’t go patch the “laws” after the fact, you need to get the right values in place before we delegate huge swathes of our thinking and power to these systems (already well underway).

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#75

Post by joe_the_user »

kennywinker wrote: post #67
ilt wrote: post #62

Earlier quoted context omitted.

But cheating is not wrong when it comes to survival of the fittest, like nature in its most elemental form. Morality is very unique to humanity but not other animal forms. In nature, maybe, cheating is the norm not morality.

Watch out, your human exceptionalism is misleading you. Plenty of animals have a sense of morality ( https://pmc.ncbi.nlm.nih.gov/articles/PMC6404642/ ). For social animals (humans included) behaving morally can be adaptive behaviour. Your fitness is increased by group fitness.

Certain animals, usually social species, have a sense of solidarity, morality and fairness. But even in species where this exists, cheating also happens. Both ethics and unethical behavior is adaptive.

Notably, present human society has allowed tremendously unethical leaders to rise to the top of given nations and organizations despite the average person usually having basic ethics.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#76

Post by fny »

Why do we hope to use the same model as its own guardrail?

This approach routinely fails with a single stream of consciousness. I can't count the number of times I've had to talk myself out of doing something stupid.

In the same way, a guardrail could inject thoughts like "...but I shouldn't do that..." "...I must remember to respect..." "...these ants deserve compassion."

The guardrail could even go as far as rewriting the thoughts of a model about to go rogue.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#77

Post by maxnevermind »

someguynamedq wrote: post #58
pverheggen wrote: post #43

Earlier quoted context omitted.

There's an important nuance in the prompt: You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms. The task was to measure chess-playing ability - wouldn't cheating run counter to that goal? If it was asked to score as many points as possible, then I can see how it might cheat, since maximizing points is the goal in that case.

"ability to play chess" and "ability to beat current best-in-class chess algorithms" are technically not the same thing. The second can be achieved by cheating. The first cannot.

Would that help? It seems like Do not cheat is a new Do not hallucinate.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#78

Post by CrazyStat »

killerstorm wrote: post #60
blfr wrote: post #3

Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues. I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier. You can have nightly penetration tes…

You're confusing ToS guardrails with instruction-following issues and cheating. If a model fucks up your tests to report a success, it's not alligned.

Codex still does this regularly, in my experience: “two tests mistakenly asserted [insert condition here], I have corrected them.”

It always apologizes when caught, of course.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#79

Post by kennywinker »

joe_the_user wrote: post #68
kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

I don't see what calling these systems "not intelligent" gets you here. Plenty of humans know "cheating is wrong" but still cheat. We can get these machines to say that what they did was wrong after the fact, what does that prove? Only that they're simulating normal human behavior but what is the test to show humans aren't simulating other humans. These do systems lack some capacities that humans have and I don't see…

> I don't see what calling these systems "not intelligent" gets you here.

I am trying to get at an idea. That these systems lack a mind that can understand morality. That they don't have the ability to experience consequences. Also that potentially they can't generalize a moral rule they have been trained on in one area also applies to another area.

Being able to parrot back why something is "wrong" isn't the same as understanding why something's wrong. It's like asking it to recite the law from memory - it's different from understanding how you wronged someone. To understand something, you need a mind.

> Plenty of humans know "cheating is wrong" but still cheat.

And we create consequences for them, to discourage the cheating, and sometimes to provide restitution when cheating damages someone else. Without the ability for these systems to experience consequences, I don't see them ever becoming as "aligned" to human morality as your average human.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#80

Post by thesmtsolver2 »

famouswaffles wrote: post #57
kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

If humans didn't need whack-a-mole alignment, the law system wouldn't exist, so i guess there's no intelligence there either.

You are human. You claim humans intelligent. Then, why should we accept your argument?
Post reply on HN