Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

schema-harness.github.io

Post reply on HN
Print view View original

61–70 of 86 posts

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#61

Post by markisus »

It's funny that the machine intelligences that arose out of the bitter lesson cracked this benchmark by meticulously modeling each individual case with rule based approaches.

When humans tried making AI through rule based approaches, maybe we only failed because we couldn't type out the rules fast enough. Or we got burned out after writing the thousandth heuristic to fix the never ending edge cases. And we also got side tracked by concerns like maintainability, modularity, and code re-use. But I wonder if we take modern or near-future LLMs who never tire and write code faster than any human, could we make a frontier level GOFAI agent?

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#62

Post by Jensson »

Ancalagon wrote: post #58
ClassAndBurn wrote: post #13

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required. Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

How is custom engineering the tooling from scratch for every task the best path forward? There will always be some engineered tools that are better than others even when re-made by the same models - not to mention the cost of starting from scratch every single time just seems wasteful with current token spend. Does "don't re-invent the wheel" not apply to agents for some reason?

> Does "don't re-invent the wheel" not apply to agents for some reason?

Correct, the purpose of agents is to become the ultimate wheel so we humans can stop needing to reinvent the wheel. For it to do that it needs to know how to reinvent wheels.

And, just to be clear, "don't re-invent the wheel" is just for small teams, as a whole humanity needs to re-invent the wheel all the time to adapt to new situations. For every wheel your team shouldn't reinvent some other teams main job is to reinvent that wheel.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#63

Post by fnordpiglet »

Tadpole9181 wrote: post #33
fnordpiglet wrote: post #30

Earlier quoted context omitted.

A great deal of mathematics is transforming nonlinear problems into linear ones and solving them with linear techniques. Others are solving non linear problems through stochastic methods. In almost all cases most non trivial math is done by transforming a harder problem into a simpler one. I get what you mean in terms of testing the model itself to see its improvement in some domain. However if you can transform the…

To quote the people who make it: > ARC-AGI-3 is an interactive reasoning benchmark which challenges AI agents to explore novel environments, acquire goals on the fly, build adaptable world models, and learn continuously. This harness does nothing to actually accomplish those goals. It's a clever trick, sure, but you aren't allowed to use a calculator on your basic algebra tests in school for a reason.

And this demonstrates this benchmark does not necessitate achieving those goals to achieve a perfect score. You seem to miss the point that almost all of math, physics, computer science is built on constructing an objective then cheating to attain it, and that demonstrates that there is an equivalency. Maybe the benchmark is flawed, or maybe the goals are not strictly necessary to attain.

For instance, is solving a math proof by enumerating all permutations exhaustively on a computer cheating? Does it matter that it is not a proof by construction? That its not descriptive? Of course not. The proof of the four color theorem is all that’s necessary and sufficient to prove it. Calculator at an algebra exam? Who cares. This isn’t an exam, this is the real world. The fact an AI can use a physics harness to perfectly achieve ARC-AGI-3 without attaining those goals demonstrates the power of the technique and that the goals are not necessary for that class of problems. Then find another benchmark that actually demands the goals be necessary and sufficient to achieve the benchmark goals. But don’t denigrate the fact we have technology today that yesterday was a fantasy.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#64

Post by dezgeg »

stared wrote: post #7

In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/ A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.

I've been trying this too: https://www.youtube.com/watch?v=brkP58pZ23w&list=PL8C_UWcLmv...

Cool to see others trying Baba Is You too.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#65

Post by Zababa »

vessenes wrote: post #14

To be clear, we’ll want to see how this performs against the hold-out set. If it holds up, though, it’s a big deal, and kind of in line with the vibes this year, which I’d typify as ‘harness matters’. Maybe we’d upgrade to ‘harness matters immensely’ if this can 100% ARC-AGI-3 on existing models (more in the 13% range without this harness). I’m pretty excited to see what sort of generalization we come to over the nex…

I think I'd typify it as "ARC-AGI doesn't matter" more than "harness matters". Or maybe "harness matters for some very specific tasks".

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#66

Post by Zababa »

Ancalagon wrote: post #58
ClassAndBurn wrote: post #13

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required. Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

How is custom engineering the tooling from scratch for every task the best path forward? There will always be some engineered tools that are better than others even when re-made by the same models - not to mention the cost of starting from scratch every single time just seems wasteful with current token spend. Does "don't re-invent the wheel" not apply to agents for some reason?

"don't reinvent the wheel" isn't a law of physics and I think is mostly said by people that never designed anything with wheels.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#67

Post by scotty79 »

stared wrote: post #7

In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/ A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.

With this harness or without?

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#68

Post by scotty79 »

ubermon wrote: post #22

> Both scores come from a fixed fallback rule: Opus 4.8 and Sol xhigh run first; games scoring below 80 are rerun with Fable 5 and Sol max, respectively, and the higher per-game score is retained. hmm, this is like pass@n until you get the high watermark? How would this mean anything?

How do humans get their results at these tasks? Don't they try until they succeed and claim their best run as their result?

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#69

Post by scotty79 »

teravor wrote: post #17
UltraSane wrote: post #15

Earlier quoted context omitted.

The simulator the model builds is comparable to the mental model of the game humans create. It is also much more efficient, GPT 5.6 Sol cost $25,000 to run on ARC-AGI-3

> simulator the model builds is comparable to the mental model of the game humans create then they should try to use that for a more complicated game than Arc AGI. Arc games are simple by design, if you have the model simulate them they become trivial.

similarly to how they are trivial for a human?

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#70

Post by YeGoblynQueenne »

Tadpole9181 wrote: post #31
daytonellwanger wrote: post #9

Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?

This is not actually running the Arc-AGI-3 anymore. To summarize TFA: 1. The AI plays the game and records outputs. 2. The AI does TDD using those outputs to create its own copy of the game. 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human. 4. The AI then runs simulated m…

>> 2. The AI does TDD using those outputs to create its own copy of the game.-

>> 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.

In order to make a copy of the game "the AI" needs to first learn the rules of the game, otherwise the copy is not going to be very faithful and therefore not very useful.

Conversely if the copy is faithful to the game... that's it. The system has learned the rules of the game. As demonstrated by its ability to code a faithful copy of the game. Otherwise, how can it create a copy of the game?

>> 4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.

That's precisely what it means to plan with a world model. The copy of the game is a world model and the system is predicting the outcome of its moves in the real game by first trying them out in the copy (i.e. "in its head"). That's exactly how, e.g. PDDL planning works: a planner has a model of an environment given in PDDL and it can try out its actions in that model, and choose an optimal sequence of actions to arrive at a goal state from an initial state, by observing the outcomes of its actions in the model.

It sounds like this "cheating" harness is actually doing something very reasonable and not cheating at all.

Have the ARC people (Chollet and friends) claimed that this result is cheating?

Post reply on HN