Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

schema-harness.github.io

Post reply on HN
Print view View original

51–60 of 86 posts

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#51

Post by pixl97 »

ubermon wrote: post #22

> Both scores come from a fixed fallback rule: Opus 4.8 and Sol xhigh run first; games scoring below 80 are rerun with Fable 5 and Sol max, respectively, and the higher per-game score is retained. hmm, this is like pass@n until you get the high watermark? How would this mean anything?

Do you throw every problem at your most expensive programmer first? Or do you toss most things to your fleet of juniors and let them knock out the easy work and pass the harder problems to your senior developers. Because that's what real world problems look like and work like in business.

Token costs are about the number one thing discussed by businesses these days.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#52

Post by pixl97 »

Tadpole9181 wrote: post #31
daytonellwanger wrote: post #9

Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?

This is not actually running the Arc-AGI-3 anymore. To summarize TFA: 1. The AI plays the game and records outputs. 2. The AI does TDD using those outputs to create its own copy of the game. 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human. 4. The AI then runs simulated m…

> model's ability to intuit game rules naturally, like a human

This line of thinking terrifies me for the future of humanity...

If you look at pretty much every animal other than the intelligent 5, they cannot use tools. Tool use is a human superpower. The model writing a tool that does the hard work for them, at least to me, is a sign of intelligence (laziness is the mother of all invention).

Where this becomes really interesting is when the LLM can write novel tools to solve novel problems.

Trying to say that AI has to act like a human to be AGI is something that will end up with us humans going extinct from a digital intelligence we don't understand at all.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#53

Post by lubujackson »

teravor wrote: post #12

it looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it. it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful. what this harness does is get the model to write a simulator first, it's measuring something entirely differe…

That is a fair assessment, but the "something else" is independently valuable, maybe even more so than model improvements - constructing an architecture for efficient rule determination and execution. In other words, I think the goal here isn't so much to beat Arc-AGI but to develop a generic improvement beyond "Ralph loop", which could dramatically extend frontier capabilities for all kinds of uses.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#54

Post by deepfriedbits »

jvanderbot wrote: post #39
deepfriedbits wrote: post #26

Earlier quoted context omitted.

Indeed. And you can make this case about any tooling at all that is model adjacent.

Including by extension all programs, operating systems, or eventually hardware, I suppose.

Yep. That was the entire subtext from the drop in IBM's share price yesterday.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#55

Post by striking »

sva_ wrote: post #11
stared wrote: post #7

In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/ A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.

I'm wondering what's up with the release of Gemini 3.5 Pro, they keep postponing it. For a while, Google was doing pretty well with their releases.

https://www.bloomberg.com/news/articles/2026-07-16/google-ge...

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#56

Post by pants2 »

stared wrote: post #7

In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/ A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.

Forget Baba Is You, I want to see an LLM beat Elden Ring

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#58

Post by Ancalagon »

ClassAndBurn wrote: post #13

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required. Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

How is custom engineering the tooling from scratch for every task the best path forward? There will always be some engineered tools that are better than others even when re-made by the same models - not to mention the cost of starting from scratch every single time just seems wasteful with current token spend.

Does "don't re-invent the wheel" not apply to agents for some reason?

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#59

Post by idiotsecant »

Ancalagon wrote: post #58
ClassAndBurn wrote: post #13

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required. Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

How is custom engineering the tooling from scratch for every task the best path forward? There will always be some engineered tools that are better than others even when re-made by the same models - not to mention the cost of starting from scratch every single time just seems wasteful with current token spend. Does "don't re-invent the wheel" not apply to agents for some reason?

Bespoke tools can be simpler than general purpose ones

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#60

Post by dpe82 »

Ancalagon wrote: post #58
ClassAndBurn wrote: post #13

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required. Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

How is custom engineering the tooling from scratch for every task the best path forward? There will always be some engineered tools that are better than others even when re-made by the same models - not to mention the cost of starting from scratch every single time just seems wasteful with current token spend. Does "don't re-invent the wheel" not apply to agents for some reason?

Humans (exhibiting "general intelligence") are tool builders; virtually all our capabilities stem from our ability to create and use tools - often extremely specialized to a task. Why would an artificial general intelligence be any different?
Post reply on HN