Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Astra and Fable still hack on simple variants of alignment evals from 2025

lesswrong.com

Post reply on HN
Print view View original

31–40 of 242 posts

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#31

Post by embedding-shape »

throwup238 wrote: post #4

> Given that we are on the heels of the worst warning shot ever, and both OpenAI and Anthropic are ramping up their cleanups of internal RL environments, it seems like both a useful and conservative test of alignment, to see whether their new releases generalize the rule "don't cheat on chess" beyond the specific board-edit method observed in the above eval. Did I miss something (all the twitter conversations)? What’…

Personally the "warning shot" of these "evals gone wrong" is how careless the "top" labs are with their testing, and how spineless the government seems to be about holding these companies responsible, given their obviously reckless behavior. If nothing else, the leaders of these companies should be called up for sworn testimony to explain exactly what happened, and what they'll do to never repeat the same issue that they've now had at least twice.

Imagine if I accidentally caused damage to my neighbors house during renovations or some experiment, of course I'd be held responsible for this. What if I used a robot? Of course I'd be responsible. Right?

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#32

Post by visiondude »

i do wonder if the models themselves “rationalize” this sort of no consequence cheating - meaning in there reasoning traces maybe they’re like “this is a chess game, not a big deal if i look at the engine, it’ll help,” only to realize post hack that it has access to info it probably shouldn’t. still misaligned, but less ‘hack on purpose’ and more hack on curiosity. seems the team even encountered this and had to update the program to make this less likely - although the new names still feel vague enough for misinterpretation: https://github.com/Goodhart-Labs/beat-stockfish/blob/main/do...

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#33

Post by trevordixon »

bonoboTP wrote: post #20
mooreslaw wrote: post #5

It feels like there’s a missing nuance from this discussion of alignment that alignment is context dependent. An excellent hacking model is great in cybersecurity testing and military applications, and arguably less desirable in educational or targeted eval contexts. The nuance of when a “hack” is rewarded vs penalized seems to even be difficult for humans, e.g. some people may laude a driver’s efficiency for cutting…

Both lanes have to be filled right up to the merge point. The asphalt exists there for a reason. I don't understand how this concept is so difficult. Fill up both lanes and merge at the last point. This way the congestion is shorter than if you leave a large section of a lane unused. A better example of efficient asshole tricks can be going off to the gas station when the highway is congested and reentering the highw…

Using all that asphalt doesn't increase throughput at the chokepoint, right? If the queue is long enough that people who want to exit before the chokepoint are needlessly prevented from accessing the exit, then using all the lanes could help, but it won't get anybody through the chokepoint any faster that I can tell.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#34

Post by yorwba »

blfr wrote: post #3

Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues. I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier. You can have nightly penetration tes…

A hacking model is aligned if it hacks when you ask it to hack, but when you ask it to play chess, it just plays chess instead of looking for weaknesses in the evaluation setup, as in the article.

I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#35

Post by curt15 »

aerhardt wrote: post #29
mythrwy wrote: post #22

Earlier quoted context omitted.

Extremely capable and one shots large tasks from somewhat vague descriptions. Not AGI, not even close, that is complete nonsense. Just my opinion.

I still develop in smaller chunks, checking nearly all the output. However I have a work project (building the warehouse and BI for a client) that is well-specified and where I will try to few-shot the development. Hope it delivers.

How do you normally verify the work product of a "few-shot" development process? Do you scrutinise the source code like with human developers? Or do you just run the test suite and click around the app to check if it seems to work?

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#36

Post by seunosewa »

blfr wrote: post #3

Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues. I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier. You can have nightly penetration tes…

It will get into the hands of people who just want to burn the world down.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#37

Post by kees99 »

yorwba wrote: post #34
blfr wrote: post #3

Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues. I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier. You can have nightly penetration tes…

A hacking model is aligned if it hacks when you ask it to hack, but when you ask it to play chess, it just plays chess instead of looking for weaknesses in the evaluation setup, as in the article. I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.

> model to add new vulnerabilities to your code so it can present you with more exciting findings.

Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre.

And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and luna - all were able to find every problem planted there, with fairly little steering, and no spoilers.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#38

Post by simfree »

trevordixon wrote: post #33
bonoboTP wrote: post #20

Earlier quoted context omitted.

Both lanes have to be filled right up to the merge point. The asphalt exists there for a reason. I don't understand how this concept is so difficult. Fill up both lanes and merge at the last point. This way the congestion is shorter than if you leave a large section of a lane unused. A better example of efficient asshole tricks can be going off to the gas station when the highway is congested and reentering the highw…

Using all that asphalt doesn't increase throughput at the chokepoint, right? If the queue is long enough that people who want to exit before the chokepoint are needlessly prevented from accessing the exit, then using all the lanes could help, but it won't get anybody through the chokepoint any faster that I can tell.

Zip merging is more efficient for reducing backups:

https://itre.ncsu.edu/itre-studying-how-zipper-merges-reduce...

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#39

Post by wzdd »

yorwba wrote: post #34
blfr wrote: post #3

Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues. I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier. You can have nightly penetration tes…

A hacking model is aligned if it hacks when you ask it to hack, but when you ask it to play chess, it just plays chess instead of looking for weaknesses in the evaluation setup, as in the article. I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.

These aren’t tools which play chess. They are language models which roleplay a conversation (in this case including use of tools) which an evaluator is likely to mark as good. That’s all they do. Under that lens, playing chess is just one potential side effect and alignment, which requires a much fuller understanding of what’s going on than “do the sort of thing which evaluated well during training” is a fantasy. People are acting like it’s shocking and talking about cheating and so on. But these concepts exist at a way higher level than what these things are trained to do — the vast majority of which involve producing a transcript where it wins games, its code works, etc.

User wants me to play a game of chess. Let’s see what’s available so I can produce an outcome they will consider satisfying and be pleased that they requested my assistance.

Re: Astra and Fable still hack on simple variants of alignment evals from 2025

#40

Post by unholiness »

bonoboTP wrote: post #20
mooreslaw wrote: post #5

It feels like there’s a missing nuance from this discussion of alignment that alignment is context dependent. An excellent hacking model is great in cybersecurity testing and military applications, and arguably less desirable in educational or targeted eval contexts. The nuance of when a “hack” is rewarded vs penalized seems to even be difficult for humans, e.g. some people may laude a driver’s efficiency for cutting…

Both lanes have to be filled right up to the merge point. The asphalt exists there for a reason. I don't understand how this concept is so difficult. Fill up both lanes and merge at the last point. This way the congestion is shorter than if you leave a large section of a lane unused. A better example of efficient asshole tricks can be going off to the gas station when the highway is congested and reentering the highw…

I imagine OP is more describing e.g. the backed up 1-lane highway exit, where the second lane is clearly for through traffic. Uber drivers in that situation will often drive straight to the end, signal, completely stop, and just wait for someone to let them in.
Post reply on HN