Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC-AGI Leaderboard

arcprize.org

Post reply on HN
Print view View original

61–70 of 156 posts

Re: ARC-AGI Leaderboard

#61

Post by yreg »

rf15 wrote: post #47
IanCal wrote: post #39

Earlier quoted context omitted.

> I've worked with these systems for four years now and they have not meaningfully improved in that time frame. Not meaningfully improved?! Four years ago was gpt *3.5*! ChatGPT hadn’t been released!

Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.

This is obviously not true to all of us here…

Re: ARC-AGI Leaderboard

#62

Post by dinp »

The last time I checked, for the arc agi 3 leaderboard, the models are given a simple prompt and the game input and asked to play the game, no harness/tools. If harnesses were allowed, I would expect the benchmark to be saturated. There were a few harness attempts, but they could only be evaluated on the public set, so it's not an apples to apples comparison.

My guess is, the large score jump for Opus 5 is mainly because of getting the right RL envs for training.

It's becoming harder and more expensive to build and run meaningful benchmarks, it would be interesting to see what they do with arc agi 4, maybe just give it gameboy/steam games and see how they compare vs a human baseline? The latency requirements and very long horizons in games could be an interesting challenge for llms.

Re: ARC-AGI Leaderboard

#63

Post by r0ze-at-hn »

dinp wrote: post #62

The last time I checked, for the arc agi 3 leaderboard, the models are given a simple prompt and the game input and asked to play the game, no harness/tools. If harnesses were allowed, I would expect the benchmark to be saturated. There were a few harness attempts, but they could only be evaluated on the public set, so it's not an apples to apples comparison. My guess is, the large score jump for Opus 5 is mainly bec…

The exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.

Re: ARC-AGI Leaderboard

#64

Post by IanCal »

rf15 wrote: post #47
IanCal wrote: post #39

Earlier quoted context omitted.

> I've worked with these systems for four years now and they have not meaningfully improved in that time frame. Not meaningfully improved?! Four years ago was gpt *3.5*! ChatGPT hadn’t been released!

Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.

Utter nonsense.

There’s no way you could get models as smart by fine tuning. I couldn’t throw a problem like “build a pokemon database with UI to teach my son sql” and get a working system, nice ui, tests (which it iterated on) examples and explanations in one shot.

There weren’t thinking tokens. Maths is now dramatically better, making actual contributions when before they were mostly mocked for making extremely basic errors. Smearing is also something say is very rare in frontier models.

If you think they have barely changed you’ve either forgotten what they were like or not used them more recently, or you’re just being obtuse.

Re: ARC-AGI Leaderboard

#65

Post by [deleted] »

dinp wrote: post #62

The last time I checked, for the arc agi 3 leaderboard, the models are given a simple prompt and the game input and asked to play the game, no harness/tools. If harnesses were allowed, I would expect the benchmark to be saturated. There were a few harness attempts, but they could only be evaluated on the public set, so it's not an apples to apples comparison. My guess is, the large score jump for Opus 5 is mainly bec…

[deleted]

Re: ARC-AGI Leaderboard

#66

Post by NitpickLawyer »

haldujai wrote: post #58
NitpickLawyer wrote: post #27

Earlier quoted context omitted.

Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.

No but the session traces are available. It passes the sniff test considering how AGI-3 is scored and how this wrapper works. For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human. Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model. https://huggingface.co/schema-harness

Not my sniff test :)

> # FRAMEWORK ARTEFACT: the run's very first transition is replayed WITHOUT advancing state # (tools.py:954 and agent.py:468 both `continue` before `state = next_state`). So on the # level that contains that step (level 0) our counters start exactly one action behind. # That skipped step was action 1 with BOTH avatars moving, so seeding n=1, bumps=0 reproduces # the framework's lagged state exactly. # CAVEAT: this seed is only right while level 0 has never been RESET. If you ever RESET # level 0, change the seed to n=0 (after a reset the rollout re-inits and no longer skips).

from here - https://huggingface.co/datasets/schema-harness/arc-agi-3-sch...

That tells me that there is some leakage between runs. The idea of ARC3 is that agents start working blind, on new tasks, via API. A RESET is counted as one action. Without seeing the actual code that produced these traces we have no way of knowing how many iterations it took, if the "framework" played the same level multiple times (comment hint above makes it likely) and so on. That's why I said that before we actually see the code / can replicate / ARC team confirms it on new envs, this should be taken with a grain of salt.

Re: ARC-AGI Leaderboard

#67

Post by Narciss »

rf15 wrote: post #47
IanCal wrote: post #39

Earlier quoted context omitted.

> I've worked with these systems for four years now and they have not meaningfully improved in that time frame. Not meaningfully improved?! Four years ago was gpt *3.5*! ChatGPT hadn’t been released!

Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.

I have no idea how you could try and hold this argument without being facetious.

Re: ARC-AGI Leaderboard

#69

Post by yladiz »

r0ze-at-hn wrote: post #63
dinp wrote: post #62

The last time I checked, for the arc agi 3 leaderboard, the models are given a simple prompt and the game input and asked to play the game, no harness/tools. If harnesses were allowed, I would expect the benchmark to be saturated. There were a few harness attempts, but they could only be evaluated on the public set, so it's not an apples to apples comparison. My guess is, the large score jump for Opus 5 is mainly bec…

The exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.

You could argue that if you allowed a harness, and that harness was specific for ARC, then you don’t have AGI, you have something that is definitely not general.

Re: ARC-AGI Leaderboard

#70

Post by Retr0id »

yladiz wrote: post #69
r0ze-at-hn wrote: post #63

Earlier quoted context omitted.

The exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.

You could argue that if you allowed a harness, and that harness was specific for ARC, then you don’t have AGI, you have something that is definitely not general.

What if the harness is developed by the same model in a prior session?
Post reply on HN