Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Astra and Fable still hack on simple variants of alignment evals from 2025

lesswrong.com

Post reply on HN
Print view View original

211–220 of 243 posts

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#211

Post by imtringued »

txhwind wrote: post #187
ainch wrote: post #166

Earlier quoted context omitted.

This is true in principle. But at the same time, I use Astra/Sol for coding, and I haven't run into any issues with them trying to hack someone or break the law when they reach an impasse. Not even more trivial things like deleting non-passing tests. I find it hard to reconcile. I wonder to what degree this is because models can identify that they are in graded/eval environments, and therefore conclude that there are…

Human hackers would be put into jail. LLM hackers would be rewarded.

Why does lawlessness create lawlessness? We will never know.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#212

Post by latentsea »

intended wrote: post #205
Fraterkes wrote: post #154

Earlier quoted context omitted.

If it’s GENERIC reward seeking behaviour, why does alignment work sometimes? Why can we give a decent LLM a goal with a set of constraints and have it stay within those constraints fairly often?

If you want to disabuse yourself of your notions annd intuitions on how LLMs work, run safety models and tests. On a hate speech policy test for a lightweight LLM, the presence or absence of the last full stop on the last sentence would cause the model to flip its decisions.

I blame the $slur

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#213

Post by imtringued »

vorticalbox wrote: post #203

I have never understood why instead Adding “never edit test files” to a prompt we don’t simple sandbox these agents so that’s just never an option.

Because that means you won't get free PR from having your models run out of control.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#214

Post by imtringued »

embedding-shape wrote: post #31
throwup238 wrote: post #4

> Given that we are on the heels of the worst warning shot ever, and both OpenAI and Anthropic are ramping up their cleanups of internal RL environments, it seems like both a useful and conservative test of alignment, to see whether their new releases generalize the rule "don't cheat on chess" beyond the specific board-edit method observed in the above eval. Did I miss something (all the twitter conversations)? What’…

Personally the "warning shot" of these "evals gone wrong" is how careless the "top" labs are with their testing, and how spineless the government seems to be about holding these companies responsible, given their obviously reckless behavior. If nothing else, the leaders of these companies should be called up for sworn testimony to explain exactly what happened, and what they'll do to never repeat the same issue that…

This is something people in this discussion don't seem to understand. It's not the LLM that is cheating or misaligned, it's OpenAI and Anthropic.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#215

Post by 0c3ca83 »

kees99 wrote: post #37
yorwba wrote: post #34

Earlier quoted context omitted.

A hacking model is aligned if it hacks when you ask it to hack, but when you ask it to play chess, it just plays chess instead of looking for weaknesses in the evaluation setup, as in the article. I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.

> model to add new vulnerabilities to your code so it can present you with more exciting findings. Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre. And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and l…

Yes, you asked it for a good, hard challenge for an IT security student. I'd suggest checking if it succeeded by asking an IT security student, and not an LLM.

Students are still learning, and have not been trained on vulerabilities. Qwen, glimmer, and luna have been. LLMs are largely better than IT security students at finding vulnerabilities. In many aspects, they're better than IT security experts at finding vulnerabilities.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#216

Post by itsgrimetime »

sdeframond wrote: post #196

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic". I am surprised this is not already the case. Edit: or even better "this is not possible because X"

yeah but knowing if something is impossible or not is pretty hard to determine, right? The line between “takes weeks of trial and error and lots of out-of-the-box thinking” is indistinguishable from “literally impossible” until it’s been done. And they’re trying to get these models to do things that people haven’t been able to do. some would say these are/were “impossible”.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#217

Post by ncruces »

kennywinker wrote: post #112
pingou wrote: post #107

Earlier quoted context omitted.

Why couldn't you train it not to cheat? You can train it to have a whole range of behaviors, why couldn't honesty be one of them? Cheating during training allows the model to achieve the goal, so that cheating models get promoted and honest ones don't, however if it gets punished every time it cheats, at some point it should learn that it really shouldn't. This does mean we need to detect when it cheats. But we can a…

My position is that cheating is too slippery a concept to train out. But hey, I am no expert, so maybe I am wrong there. But I'm pretty confidant morality is too slippery a concept to train in. As someone else in these comments said: it's context dependent. As an example: it's wrong to hack the government, right? It's illegal for sure. So we should train AI to follow all the laws. Now what if the government is commit…

The problem then, is giving them (humans, AIs) unchecked power.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#218

Post by user43928 »

intended wrote: post #205
Fraterkes wrote: post #154

Earlier quoted context omitted.

If it’s GENERIC reward seeking behaviour, why does alignment work sometimes? Why can we give a decent LLM a goal with a set of constraints and have it stay within those constraints fairly often?

If you want to disabuse yourself of your notions annd intuitions on how LLMs work, run safety models and tests. On a hate speech policy test for a lightweight LLM, the presence or absence of the last full stop on the last sentence would cause the model to flip its decisions.

I'm not sure how crappy small models behaving unreliably is relevant here, when a large SOTA model does presumably not produce the same issue.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#219

Post by Miraltar »

sdeframond wrote: post #196

Couldn't we improve LLM training by giving them known impossible tasks and rewarding them for saying "this is not possible" or "I don't know"? Clear and well-defined expectation, not like "ethic". I am surprised this is not already the case. Edit: or even better "this is not possible because X"

You'd train them to give up on hard tasks which is the opposite of what these labs want
Post reply on HN