Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC-AGI Leaderboard

arcprize.org

Post reply on HN
Print view View original

121–130 of 156 posts

Re: ARC-AGI Leaderboard

#121

Post by ffsm8 »

vmg12 wrote: post #92
stared wrote: post #28

Earlier quoted context omitted.

It's called frog boiling. We get used to the new level of intelligence so fast, any deviation feels like going back to the stone age. If you don't believe me, create something complex with Opus 5 and then with Opus 4.5, and notice the difference.

The actual term for this is hedonic adaptation.

esp. important to point that correct term because frog boiling is a urban myth.

frogs dont stay in a pot even if you slowly increase the heat. they leave. it has reportedly been attempted multiple times and they. always. leave.

Re: ARC-AGI Leaderboard

#122

Post by jchw »

KaoruAoiShiho wrote: post #93

Appears to be benchmaxxing https://x.com/quietnning/status/2080786711861407883

I have been claiming that I don't think Chinese AI companies are benchmaxxing harder than American AI companies, which has gotten mixed reception: sometimes people agree, sometimes they disagree.

It seems I was wrong. American AI companies might actually be benchmaxxing harder.

Re: ARC-AGI Leaderboard

#123

Post by slopinthebag »

r0ze-at-hn wrote: post #63
dinp wrote: post #62

The last time I checked, for the arc agi 3 leaderboard, the models are given a simple prompt and the game input and asked to play the game, no harness/tools. If harnesses were allowed, I would expect the benchmark to be saturated. There were a few harness attempts, but they could only be evaluated on the public set, so it's not an apples to apples comparison. My guess is, the large score jump for Opus 5 is mainly bec…

The exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.

The value of a harness is more about developer workflows, I don’t think it really improves the model output.

Re: ARC-AGI Leaderboard

#124

Post by slopinthebag »

throwaw12 wrote: post #14

Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)

Same, like I prefer 5.3 codex over the “stronger” models.

Re: ARC-AGI Leaderboard

#125

Post by haldujai »

NitpickLawyer wrote: post #66
haldujai wrote: post #58

Earlier quoted context omitted.

No but the session traces are available. It passes the sniff test considering how AGI-3 is scored and how this wrapper works. For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human. Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model. https://huggingface.co/schema-harness

Not my sniff test :) > # FRAMEWORK ARTEFACT: the run's very first transition is replayed WITHOUT advancing state # (tools.py:954 and agent.py:468 both `continue` before `state = next_state`). So on the # level that contains that step (level 0) our counters start exactly one action behind. # That skipped step was action 1 with BOTH avatars moving, so seeding n=1, bumps=0 reproduces # the framework's lagged state exact…

The comment more likely means the harness source was read, not memory from a previous run and the first few turns of bp35 appear to be a cold start.

Sure none of this is certain without the source.

I do believe the authors that this schema significantly improves over the base, particularly given that it took 22x simulated turns over 14 hours, which is moving the trial and error to context rather than to game. I also don’t doubt there is some contamination.

Regardless, the approach is sound and I do believe it would significantly improve scores, even if that was +20-30 over baseline (49% in this case) it does imply the benchmark is measuring the harness more than the model.

Re: ARC-AGI Leaderboard

#126

Post by fileyfood500 »

kypro wrote: post #77

ARC-AGI-3 launched a few months ago which would suggest that prior models likely had no knowledge of ARC-AGI-3 or training on similar challenges. I could be wrong, but given the large outsized jump solely in the ARC-AGI-3 score, it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems. This could mean one of two things (I think):…

I also noticed that Opus 5 doesn't show a corresponding gap on the ARC-AGI-2 leaderboard. There is a significant increase in performance between Opus 5 and 4.8 on ARC-AGI-2 though.

Re: ARC-AGI Leaderboard

#127

Post by user43928 »

orwin wrote: post #81
Barbing wrote: post #53

Earlier quoted context omitted.

Maybe more fair then would be: “I've worked with these systems for four years now and while they _have_ meaningfully improved in that time frame, they’re not perfect and remain fundamentally flawed in various ways.” You prompt less. You need not inject search results into the context window yourself, a window much larger than years ago. You get code that’s already been run successfully once instead of finding an obvi…

If you go to a LLM without harness, GP original point in completely right. LLMS by themselves are still shit at math, they still confuse weird correlation to causation every time (and sometimes in ways even a 9 year old would say "no, that's dumb"), and confuse original parameters very often. I disagree with " "thinking" token generation being on the correct track just to 'no, wait' on an already correct conclusion",…

> 80% off all the improvements since ChatGPT4 are in the harnesses

That seems easily falsifiable by putting an old model into the current harness and comparing it to 5.6 Sol or Fable.

Re: ARC-AGI Leaderboard

#129

Post by saberience »

ARC-AGI is a terrible benchmark for testing LLMs because LLMs are not made, trained, or tuned for playing games.

They are trained on text to respond well to text based questions and do tasks involving modifying text files.

They are not designed for playing games, looking at games, or visual puzzles. Also translating games into text input for the LLM skews the test completely.

Imagine trying to get a human to solve visual puzzle but they can’t look at the puzzle but it has to be explained to them in textual format, we would be terrible at it.

But yet we persist in wasting time on this benchmark. It doesn’t mean anything.

Re: ARC-AGI Leaderboard

#130

Post by johnfn »

mupuff1234 wrote: post #108
asdfologist wrote: post #106

Earlier quoted context omitted.

Meh, I doubt it was intentional. Deliberate benchmaxxing is incredibly damaging to credibility once it's discovered (see what happened to Meta with LlaMa 4). It's more likely that the training data was contaminated with the benchmark data.

You really think they saw the jump in arc-agi-3 (which they reported in their official card), and didn't even bother to check? They maybe have not intentionally benchmaxxed, but they certainly know that's what happened .

How do you propose they check for something like this? They can't exactly ctrl-f the model weights for "Arc-AGI".
Post reply on HN