Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC-AGI Leaderboard

arcprize.org

Post reply on HN
Print view View original

51–60 of 156 posts

Re: ARC-AGI Leaderboard

#51

Post by Barbing »

rf15 wrote: post #45
fakwandi_priv wrote: post #38

Earlier quoted context omitted.

> - Math completely fails in longer contexts Not sure what longer contexts we're talking about but didn't we have an old math problem optimized, which even the LLM itself was surprised about, just a week ago? Something which wasn't possible 6 months ago.

I mean calculations, not mathematical proofs

If they use Python to fill the gap, and the end user doesn’t have to know or care, is it unfair to assess this as progress and attribute the progress to the _system_?

OK, the core technology that is the language model still can’t math as well as you’d hope, but how about the end result users see from the system when they interface with it?

“Did you know humans are better at flying today than they were a thousand years ago?” ‘No they’re not, they need planes.’ Technically correct in a way but isn’t it kind of annoying to be so stubbornly pedantic when the context is speed of reaching Point B from Point A?

Re: ARC-AGI Leaderboard

#52

Post by NitpickLawyer »

Stevvo wrote: post #43
NitpickLawyer wrote: post #27

Earlier quoted context omitted.

Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.

If you stop and think about the problem it really is quite simple. Just need to build a graph of the game state and then run A* to get to the end.

You really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe.

Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them.

The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session.

Seriously, play some of the games. They're fun.

Re: ARC-AGI Leaderboard

#53

Post by Barbing »

mdp2021 wrote: post #48
Barbing wrote: post #46

Earlier quoted context omitted.

Like how toddlers’ skills don’t meaningfully improve on infants’, because either could wake up in a wet bed.

Let us be more clear: there is no structural jump, no architectural overcoming of the original fault. (Edit: and on a similar point, structural properties such as having static ntetworks, as opposed to continuously learning and improving architectures (such as us), will reveal that there is still road ahead.)

Maybe more fair then would be: “I've worked with these systems for four years now and while they _have_ meaningfully improved in that time frame, they’re not perfect and remain fundamentally flawed in various ways.”

You prompt less. You need not inject search results into the context window yourself, a window much larger than years ago. You get code that’s already been run successfully once instead of finding an obvious show stopping bug yourself.

The technology is not a brand new one that fixed everything wrong with the old one, no, but not sure I would’ve noticed your comment if it had been such a bland observation. I genuinely assume good faith here… will say am tempted to assume the standards of someone posting such a thing might be impossibly high. Glad to be having a fun conversation instead of getting your grades on my work product or something :)

Re: ARC-AGI Leaderboard

#55

Post by virgildotcodes »

rf15 wrote: post #47
IanCal wrote: post #39

Earlier quoted context omitted.

> I've worked with these systems for four years now and they have not meaningfully improved in that time frame. Not meaningfully improved?! Four years ago was gpt *3.5*! ChatGPT hadn’t been released!

Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.

> you could do the same back then with multiple models and more targeted finetuning

I mean, come on, this is just not true. You could not achieve anything like what you can with modern agentic coding with Fable / 5.6 Sol from any combination or configuration of GPT 3.5 era models.

It's like saying that a teenager isn't an intellectually meaningful improvement over a toddler.

Sure they're both still fundamentally flawed humans prone to cognitive error, but one is clearly more likely to hit the mark than the other when assigned a task.

Re: ARC-AGI Leaderboard

#56

Post by danielbln »

anonzzzies wrote: post #49
rf15 wrote: post #47

Earlier quoted context omitted.

Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.

> you could do the same back then with multiple models and more targeted finetuning Are you one of those anonymous billionaires as if you did this a few years ago, you would've been famous and rich.

OP is delusional or deliberately optuse. I work in the space and stare down these systems 12h/day, and saying the systems haven't meaningfully improved is ludicrous.

Re: ARC-AGI Leaderboard

#57

Post by SubiculumCode »

throwaw12 wrote: post #14

Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)

5 seems incredibly smart to me in my conversations today about some pretty niche ideas in.longitudinal modeling. It.felt.like a big step up.from 4.8, to me

Re: ARC-AGI Leaderboard

#58

Post by haldujai »

NitpickLawyer wrote: post #27
haldujai wrote: post #23

Earlier quoted context omitted.

For frontier models, not local. https://schema-harness.github.io/

Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.

No but the session traces are available. It passes the sniff test considering how AGI-3 is scored and how this wrapper works.

For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human.

Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model.

https://huggingface.co/schema-harness

Re: ARC-AGI Leaderboard

#59

Post by Marha01 »

rf15 wrote: post #47
IanCal wrote: post #39

Earlier quoted context omitted.

> I've worked with these systems for four years now and they have not meaningfully improved in that time frame. Not meaningfully improved?! Four years ago was gpt *3.5*! ChatGPT hadn’t been released!

Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.

> you could do the same back then with multiple models and more targeted finetuning

Definitely not, lol.

Post reply on HN