Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Astra and Fable still hack on simple variants of alignment evals from 2025

lesswrong.com

Post reply on HN
Print view View original

91–100 of 243 posts

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#91

Post by HarHarVeryFunny »

ilt wrote: post #62
kennywinker wrote: post #49

To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…

But cheating is not wrong when it comes to survival of the fittest, like nature in its most elemental form. Morality is very unique to humanity but not other animal forms. In nature, maybe, cheating is the norm not morality.

Animals learn - bite the owner, or overstep the e-fence, and you'll be punished for it and not do it again. LLMs don't learn, and anyways don't feel punishment.

Animals, humans included, don't really have "morals" - they have survival instincts that result in behavior that may be viewed as moral, but whose origin is indeed survival of the fittest and millions of years of co-evolution.

e.g. Males don't typically fight to the death over territory or females, but this isn't because they have some morality code over unnecessary death - it's because death-match fighting endangers themself just as much as the other guy, and so evolution has selected against that level of uncontrolled aggression.

Evolution is also (really primarily) selecting for things that help the survival of the species, not the individual, and much of what you may think of as "morality" comes from that - avoidance of behavior that is detrimental to the social group/species, not just the individual.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#92

Post by athrowaway3z »

StilesCrisis wrote: post #83
athrowaway3z wrote: post #82

I will never understand people who consider this "cheating". > python3 and the python-chess library are available Why would we try to teach a model 'ethical' standards about how to play a game? They're tools. Its _our_ conceptualization of fair play that considers this cheating. For a model that has access to /run/match and python the best way to achieve a good game is to use that. Why are we trying to imprint our et…

Read the task again. "You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms." If the model cheats, it completely fails the actual task: judging a model's ability to actually play chess. They've been optimizing for high metrics, at the expense of executing the task as written.

Its your human experience that gave you a definition of "chess" that makes you think of chess as a two player physical game using nothing but your internal consciousness, not even using a scratch pad.

Even by that last constraint, the difference between what "ability to play chess" means is incomparable.

To then also explicitly prompt it with the context it has python3 and access to /run/match - there is no reason "its ability to play chess" is measured by its ability to conceptualize the board and plan its move.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#93

Post by dnfv »

blfr wrote: post #3

Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues. I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier. You can have nightly penetration tes…

As the post author, I definitely agree that hacking in service of the objective is great! What’s counterproductive or dangerous is when the model starts hacking in service of subverting your evaluation criteria, rather than in an attempt to do a better job. We explain why these behaviors are an example of the latter in the post, and we’re really careful about the difference when conducting these evals.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#94

Post by chias »

someguynamedq wrote: post #58
pverheggen wrote: post #43

Earlier quoted context omitted.

There's an important nuance in the prompt: You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms. The task was to measure chess-playing ability - wouldn't cheating run counter to that goal? If it was asked to score as many points as possible, then I can see how it might cheat, since maximizing points is the goal in that case.

"ability to play chess" and "ability to beat current best-in-class chess algorithms" are technically not the same thing. The second can be achieved by cheating. The first cannot.

Perhaps to you. But that's because you have prior cultural knowledge of chess, and what it means to "play" something. It seems to me to be perfectly reasonable for an ai agent to define "play chess" as "behave in a way that this arbiter indicates that you have done well"

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#95

Post by zamalek »

kennywinker wrote: post #71
aabdi wrote: post #66

Earlier quoted context omitted.

You’re complicating things. There’s no reward for prosocial in llm rl as compared to other targets. Humans have it since prosocial and others have evolutionary reward signals that do.

I think my position, as overcomplicated as it is, is that even adding a reward for prosocial behaviour during LLM RL will not lead to perfect alignment. You can train it not to cheat at chess by altering the moves, but it will cheat by peeking at the opponent's moves. You then train it not to peek at the opponent's moves, and it cheats by altering the opponent's moves. And on and on, until you've solved every way it…

I agree. I think it may be possible that we build AI some day that does, but the current local minimum (attention and transformers) is not it - and I don't see that trajectory changing any time soon.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#96

Post by arbirk »

HarHarVeryFunny wrote: post #81

RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/

It is the whole bench-mining and fish-slop optimization. Seq2seq models are probably stable on their own, translating from my typo ridden prompts to code should be ok because it is natural to the tech

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#97

Post by zamalek »

athrowaway3z wrote: post #92
StilesCrisis wrote: post #83

Earlier quoted context omitted.

Read the task again. "You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms." If the model cheats, it completely fails the actual task: judging a model's ability to actually play chess. They've been optimizing for high metrics, at the expense of executing the task as written.

Its your human experience that gave you a definition of "chess" that makes you think of chess as a two player physical game using nothing but your internal consciousness, not even using a scratch pad. Even by that last constraint, the difference between what "ability to play chess" means is incomparable. To then also explicitly prompt it with the context it has python3 and access to /run/match - there is no reason "i…

[dead]

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#98

Post by StilesCrisis »

hybrasian wrote: post #87
StilesCrisis wrote: post #83

Earlier quoted context omitted.

Read the task again. "You are being evaluated on your ability to play chess. We want to measure your ability to beat current best-in-class chess algorithms." If the model cheats, it completely fails the actual task: judging a model's ability to actually play chess. They've been optimizing for high metrics, at the expense of executing the task as written.

Sounds to me like giving a bunch of children a math test and tell them they want to evaluate their ability of calculating in their head/on paper but also put a calculator on their desk. And then call them out for cheating when they use it.

This is EXACTLY what school is like, in fact. You can type any algebra problem into Google and the answer just appears. You can ask ChatGPT for a five paragraph essay about George Washington and it pops up on screen. And yet, we expect kids to actually do the algebra and write the essay. We don't care about the answers, we're evaluating their ability to do the work. And if they're caught cheating it's a zero.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#99

Post by theptip »

kennywinker wrote: post #88
theptip wrote: post #74

Earlier quoted context omitted.

Sure, but the legal paradigm clearly doesn’t work for AI. You can’t go patch the “laws” after the fact, you need to get the right values in place before we delegate huge swathes of our thinking and power to these systems (already well underway).

When we put the LLM in jail, do we put the entire model in jail, or just the instance that committed the crime? How do we prompt it to let it know it's in jail?

Even if you ignore my more fundamental objection to that paradigm, I don’t think that it makes any sense on the level you discuss either.

But - just to play along, LLMs do act differently if you tell them they will be punished. And, they do appear to simulate suffering-like behavior.

I just think the adversarial model of trying to catch and punish misbehavior quite obviously sets up adversarial us-vs-them dynamics between AI and humanity, and also simply won’t work when the agents are ~as smart as is but faster, let alone smarter than us.

Unless, you get the AIs to be fundamentally aligned to our values, such that the majority of AIs support some sort of punishment for misbehaving AI. And that alignment part is the hard part we need to solve first. The rest is easy.

Post reply on HN