Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

schema-harness.github.io

Post reply on HN
Print view View original

81–86 of 86 posts

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#81

Post by Zababa »

Legend2440 wrote: post #76
Zababa wrote: post #75

Earlier quoted context omitted.

DeepSeek V3.2 was tried without reasoning and it got 57% on ARC AGI 1. It's a 7 month model, so I'm pretty confident that base LLMs would be able to solve ARC AGI 1 without reasoning/CoT.

No, you are misunderstanding the paper. https://arxiv.org/abs/2607.06764 The base model got 15%. They built an elaborate looping harness that allows it to burn 100k tokens "thinking" about the problem, which got the 57%. This is just an alternative approach to reasoning.

I'm not referring to this paper, I'm referring to this leaderboard: https://arcprize.org/leaderboard. Set it to "arc agi 1", "base LLM" and you'll see deepseek at 57%. Submitted 2025-12-01, $0.120 per task. The paper you linked was later than that, and also says "We do not report an official ARC Prize leaderboard score".

So this paper doubled the price to get the same exact result at base Deepseek 3.2 at launch, and wasn't even tested on the verified set.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#82

Post by larodi »

pixl97 wrote: post #52
Tadpole9181 wrote: post #31

Earlier quoted context omitted.

This is not actually running the Arc-AGI-3 anymore. To summarize TFA: 1. The AI plays the game and records outputs. 2. The AI does TDD using those outputs to create its own copy of the game. 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human. 4. The AI then runs simulated m…

> model's ability to intuit game rules naturally, like a human This line of thinking terrifies me for the future of humanity... If you look at pretty much every animal other than the intelligent 5, they cannot use tools. Tool use is a human superpower. The model writing a tool that does the hard work for them, at least to me, is a sign of intelligence (laziness is the mother of all invention). Where this becomes real…

It is not the model now showing intelligence. Tis the dialogue. The medium is the LLM and the harness is then a next level construct.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#83

Post by Legend2440 »

Zababa wrote: post #81
Legend2440 wrote: post #76

Earlier quoted context omitted.

No, you are misunderstanding the paper. https://arxiv.org/abs/2607.06764 The base model got 15%. They built an elaborate looping harness that allows it to burn 100k tokens "thinking" about the problem, which got the 57%. This is just an alternative approach to reasoning.

I'm not referring to this paper, I'm referring to this leaderboard: https://arcprize.org/leaderboard . Set it to "arc agi 1", "base LLM" and you'll see deepseek at 57%. Submitted 2025-12-01, $0.120 per task. The paper you linked was later than that, and also says "We do not report an official ARC Prize leaderboard score". So this paper doubled the price to get the same exact result at base Deepseek 3.2 at launch, and…

I think this is an error in the leaderboard. Looking at the test logs, they had reasoning effort set to 'high'. So it should be in the CoT category instead of the base LLM category.

https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e...

  "kwargs": {
     "max_tokens": 100000,
     "stream": true,
     "reasoning_effort": "high",
     "rate_limit": {
        "rate": 2,
        "period": 60
     }
  }
The other paper ran Deepseek v3.2 without reasoning as a baseline and got 15.5%, which is much more in line with other base LLMs like GPT-5.2.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#84

Post by notjackiechan »

ClassAndBurn wrote: post #13

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required. Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

I think there is a chasm to cross. the model's training to be aware of the harness it is in, at the same time building probes and observation tool can help it to cross that chasm.

Still seem too soon for a model to have that ability to build a harness on its own, and swap its session to another harness in the same environment. Like a snake shedding its skin, but in this case its harness.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#85

Post by Zababa »

Legend2440 wrote: post #83
Zababa wrote: post #81

Earlier quoted context omitted.

I'm not referring to this paper, I'm referring to this leaderboard: https://arcprize.org/leaderboard . Set it to "arc agi 1", "base LLM" and you'll see deepseek at 57%. Submitted 2025-12-01, $0.120 per task. The paper you linked was later than that, and also says "We do not report an official ARC Prize leaderboard score". So this paper doubled the price to get the same exact result at base Deepseek 3.2 at launch, and…

I think this is an error in the leaderboard. Looking at the test logs, they had reasoning effort set to 'high'. So it should be in the CoT category instead of the base LLM category. https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e... "kwargs": { "max_tokens": 100000, "stream": true, "reasoning_effort": "high", "rate_limit": { "rate": 2, "period": 60 } } The other paper ran Deepseek v3.2 without reasoning…

Interesting, good find! Yeah I may be wrong and this may be an error in the leaderboard. Weirdly it shows no reasoning cost and no reasoning tokens used, but for example here https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e... the answer is super short but it says "4945" completion tokens.

Re: Schema Harness Achieves ~99% on Arc‑AGI‑3 Public

#86

Post by HarHarVeryFunny »

YeGoblynQueenne wrote: post #71
Zababa wrote: post #65

Earlier quoted context omitted.

I think I'd typify it as "ARC-AGI doesn't matter" more than "harness matters". Or maybe "harness matters for some very specific tasks".

ARC-AGI is a bit of a joke at this point. The first version was supposed to be hard for neural nets because it had few examples and a super secret test set, with a different distribution than the public set no less. It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation, thanks to data augmentation techniques that increased the av…

Benchmarks are meant to measure something, so can't be too hard else all measurements will be 0. At the same time the systems being tested - LLMs - are getting larger and more capable, at least in the narrow areas most benchmarks are focusing on, so all benchmarks will continually become saturated and need to be revised.

So, are you against all benchmarks or specifically ARC AGI? At least ARC AGI is trying to test for something a bit different and not play to the text prediction strength of LLMs. It should go without saying that no single test, or type of test, can claim to test for AGI or human level intelligence, which would require a suite of tests as broad and varied as the generality of intelligence you are trying to test for.

Post reply on HN