Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

schema-harness.github.io

Post reply on HN
Print view View original

11–20 of 86 posts

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#11

Post by sva_ »

stared wrote: post #7

In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/ A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.

I'm wondering what's up with the release of Gemini 3.5 Pro, they keep postponing it. For a while, Google was doing pretty well with their releases.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#12

Post by teravor »

it looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it.

it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful.

what this harness does is get the model to write a simulator first, it's measuring something entirely different.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#13

Post by ClassAndBurn »

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required.

Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#14

Post by vessenes »

To be clear, we’ll want to see how this performs against the hold-out set. If it holds up, though, it’s a big deal, and kind of in line with the vibes this year, which I’d typify as ‘harness matters’. Maybe we’d upgrade to ‘harness matters immensely’ if this can 100% ARC-AGI-3 on existing models (more in the 13% range without this harness).

I’m pretty excited to see what sort of generalization we come to over the next 12 months on the harness side: if it turns out this can be RLed in as ‘consider if building a world model might help here’ and we get this as another native capacity, that will be interesting. If we get 100 of those problem-solving strategies all included, feels like we will see another hurdle cleared in terms of usefulness.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#15

Post by UltraSane »

teravor wrote: post #12

it looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it. it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful. what this harness does is get the model to write a simulator first, it's measuring something entirely differe…

The simulator the model builds is comparable to the mental model of the game humans create. It is also much more efficient, GPT 5.6 Sol cost $25,000 to run on ARC-AGI-3

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#16

Post by vessenes »

ClassAndBurn wrote: post #13

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required. Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

Except in this case, it isn't yet smart enough. But I agree, building this capability in is coming, and will be really awesome.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#17

Post by teravor »

UltraSane wrote: post #15
teravor wrote: post #12

it looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it. it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful. what this harness does is get the model to write a simulator first, it's measuring something entirely differe…

The simulator the model builds is comparable to the mental model of the game humans create. It is also much more efficient, GPT 5.6 Sol cost $25,000 to run on ARC-AGI-3

    > simulator the model builds is comparable to the mental model of the game humans create
then they should try to use that for a more complicated game than Arc AGI. Arc games are simple by design, if you have the model simulate them they become trivial.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#18

Post by vessenes »

teravor wrote: post #12

it looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it. it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful. what this harness does is get the model to write a simulator first, it's measuring something entirely differe…

This is classic goalpost movement. Arc-AGI-3 was launched this year with roughly 0.5% success for frontier models. being able to 99% it in less than six months sets a new record for Arc-AGI saturation timeline. Speaking of singularity measures. It is definitely a big deal, not least in that Chollet needs to cancel his summer vacation and write Arc-AGI-4 now.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#19

Post by teravor »

vessenes wrote: post #18
teravor wrote: post #12

it looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it. it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful. what this harness does is get the model to write a simulator first, it's measuring something entirely differe…

This is classic goalpost movement. Arc-AGI-3 was launched this year with roughly 0.5% success for frontier models. being able to 99% it in less than six months sets a new record for Arc-AGI saturation timeline. Speaking of singularity measures. It is definitely a big deal, not least in that Chollet needs to cancel his summer vacation and write Arc-AGI-4 now.

Arc AGI are simple games, the hardness comes from the input being basically adversarial to LLM training. if you use an LLM scaffold that removes the adversarial part you are measuring something else.

the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes. this would actually be impressive if you got it to do that for a much more complicated game than Arc.

with this harness the ARC AGI test becomes a test of whether or not the model can work out the transition rules in a very simple game.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#20

Post by ClassAndBurn »

vessenes wrote: post #16
ClassAndBurn wrote: post #13

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required. Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

Except in this case, it isn't yet smart enough. But I agree, building this capability in is coming, and will be really awesome.

It's likely smart enough. It just needs to be told to do it and provided the ability to introspect it. How close could foundational models get to building this harness if explicitly prompted to?

We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.

Post reply on HN