Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
schema-harness.github.io
Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
1–10 of 86 posts
Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
#2> Schema, the harness we introduce today, reaches 99% on the ARC‑AGI‑3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol.
Impressive results. Will this translate to coding agents (and training general purpose and for coding LLMs) too?
---
> When Michelson and Morley could not detect the medium light was supposed to wave in, Lorentz took the first route: keep the aether, patch the rules with contraction hypotheses that absorbed the null result. Einstein took the second: in special relativity, he discarded the aether as part of the state and made simultaneity frame-relative, yielding a simple electrodynamics of moving bodies.
BECs, SVT, Superfluid Quantum Gravity
Massful photons are modeled with Proca fields. Like Einstein, Proca was also a student of Minkowski. The Mass-Equivalence principle ~~does not~~ still holds if photons have mass.
(edit) Energy-momentum relation: https://en.wikipedia.org/wiki/Energy%E2%80%93momentum_relati...
> could not detect the medium light was supposed to wave in,
Superfluid Quantum Gravity (Fedi,) says that there is a medium that light waves through; there is not nothing in space, space is a quantum dilatant superfluid with near-zero viscosity.
Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
#3What does it mean to reach 99% score on Arc-AGI-3? That the agent is able to tackle difficult problems?
Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
#4What does it mean to reach 99% score on Arc-AGI-3? That the agent is able to tackle difficult problems?
It doesn't necessarily mean anything to reach 99% on the public set. All of the public set is known in advance, so it's possible to hardcode rules that make this easy for the models. ARC-AGI-3 is supposed to measure generalization to unseen games, so the only score that matters is the score on the held out private test set that nobody outside the ARC prize foundation has access to. Also, I believe the private set is significantly harder than the public set.
Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
#5(1) What does it score on the private test set?
(2) Does this approach generalize to, e.g., Atari or NES games, or is it just hard-coding priors about the games into the model (as Chollet specifically warned was a chronic problem in benchmarks in the original Arc-AGI paper)
Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
#6[deleted]
Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
#7In the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/
A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.
Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
#8We need to see private set results, but if this holds then it might represent a breakthrough in other domains as well.
Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
#9Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?
Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
#10Big jump for sure, but definitely comes with a giant grain of salt lacking open-sourcing the harness itself and measuring performance on the held-out set.