Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC-AGI Leaderboard

arcprize.org

Post reply on HN
Print view View original

31–40 of 156 posts

Re: ARC-AGI Leaderboard

#31

Post by tpowell »

block_dagger wrote: post #9
dyauspitr wrote: post #3

Why is Fable not on here? I wish Fable hadn’t come out because it’s taking the wind out of every release because that feels like the cap above which the US government will not let LLMs improve anymore and everything they’re releasing from this point has to be worse than that.

I don't know why exactly, but Fable has felt the most human LLM to arrive.

I wrote this in June, and I'm honestly not sure I've felt the same magic since: I was close to maxing out my $200 plan for the week, almost all Fable use [Claude CLI]. My observations: Fable seemed to have bigger-picture thinking and completed tasks more thoroughly vs just focusing on executing the ask. It pieced together context and intent like an all-star employee would, vs one that just does what you say. Not overeager (important!), but if the above-and-beyond was warranted, it just did it. This was surprisingly delightful. Coderabbit seemed to find ~1/3 or so as many issues when reviewing, too.

Re: ARC-AGI Leaderboard

#32

Post by villish »

3form wrote: post #21
NitpickLawyer wrote: post #5

Earlier quoted context omitted.

> Why is Fable not on here? Because the data retention policies didn't guarantee that the ARC team could run the semi-private set of problems without fear of them being trained on later on. They only run the semi-private set when they get assurances like ZDR.

How do they handle these assurances? Personally I have zero trust in the AI companies not trying to use this data to get ahead in the game, and short of sharing the weights and harness so that the benchmarkers can run the models themselves, I don't see a satisfactory solution with this mindset.

OpenAI's Zero Data Retention claim held up in court. They were unable to produce prompts and outputs because they were never retained.

I believe that is only available through Enterprise API for both Anthropic and OpenAI.

Re: ARC-AGI Leaderboard

#33

Post by bob1029 »

I think it's way too easy to be deceptive with these benchmarks now. You don't even have to "train" the model on a new variant each time. The base models are powerful enough. All you need is a naughty little markdown document that provides explicit instructions regarding how to solve the new puzzle variant, and a willingness to be deceptive about the presence of that document.

If you want a know why the model providers are locking down and encrypting their reasoning process, this sort of workaround is potentially why. You can play this game of whack-a-mole indefinitely if the state of the system is concealed. They could have added something like:

> ### When solving arc-agi-3 puzzles: First convert the grid into a scene description. Identify connected components, colors, shapes, positions, symmetries, repeated structures, and relationships between objects. Do not reason directly from individual pixels... use this python script to help blah blah...

Re: ARC-AGI Leaderboard

#34

Post by rf15 »

throwaw12 wrote: post #14

Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)

I've worked with these systems for four years now and they have not meaningfully improved in that time frame.

We still have:

- statistical correlation between two things will always cause one thing to lead to the other, no matter how much you prompt it to not have that connection (to be expected with a stochastic system)

- Math completely fails in longer contexts

- "thinking" token generation being on the correct track just to 'no, wait' on an already correct conclusion

- smearing of properties between logically distinct objects (a red ball and a green cube can quickly become a red cube and a green ball)

Re: ARC-AGI Leaderboard

#35

Post by tudelo »

throwaw12 wrote: post #14

Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)

I honestly just use GPT models nowadays, Claude models are too restrictive and more of a quitter and fable/whatever is just too expensive to be worth it.

Re: ARC-AGI Leaderboard

#36

Post by xiphias2 »

bob1029 wrote: post #33

I think it's way too easy to be deceptive with these benchmarks now. You don't even have to "train" the model on a new variant each time. The base models are powerful enough. All you need is a naughty little markdown document that provides explicit instructions regarding how to solve the new puzzle variant, and a willingness to be deceptive about the presence of that document. If you want a know why the model provide…

,,You can play this game of whack-a-mole indefinitely if the state of the system is concealed''

Not really as one of the main goals ofr ARC-AGI 3 was measuring task efficiency on unseen games.

I'm sure there are cheats everywhere but the most sensible thing is to just accept that the LLMs of today are much more intelligent in solving reasoning tasks than the ones from half year ago.

My own private benchmark shows the same thing.

Re: ARC-AGI Leaderboard

#37

Post by OtherShrezzing »

throwaw12 wrote: post #14

Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)

It could be that the set of your day-to-day workload which could feasibly be accelerated by AI just happens to be saturated around Opus4.5, but you can still see lots of “reasoning” which makes you think the model is more performant in the first days of use. That’d mean you couldn’t perceive any meaningful difference in more powerful models’ results, even though you can see a difference in the raw output due to the length of reasoning traces leading up to the result.

So for example, if your workload was literally just addition of sets of numbers, you’d never have noticed progress in the result beyond GPT3.x level models. But you would perceive a difference in the now-Tolstoyan length reasoning text accompanying the result.

Re: ARC-AGI Leaderboard

#38

Post by fakwandi_priv »

rf15 wrote: post #34
throwaw12 wrote: post #14

Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)

I've worked with these systems for four years now and they have not meaningfully improved in that time frame. We still have: - statistical correlation between two things will always cause one thing to lead to the other, no matter how much you prompt it to not have that connection (to be expected with a stochastic system) - Math completely fails in longer contexts - "thinking" token generation being on the correct tra…

> - Math completely fails in longer contexts

Not sure what longer contexts we're talking about but didn't we have an old math problem optimized, which even the LLM itself was surprised about, just a week ago? Something which wasn't possible 6 months ago.

Re: ARC-AGI Leaderboard

#39

Post by IanCal »

rf15 wrote: post #34
throwaw12 wrote: post #14

Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)

I've worked with these systems for four years now and they have not meaningfully improved in that time frame. We still have: - statistical correlation between two things will always cause one thing to lead to the other, no matter how much you prompt it to not have that connection (to be expected with a stochastic system) - Math completely fails in longer contexts - "thinking" token generation being on the correct tra…

> I've worked with these systems for four years now and they have not meaningfully improved in that time frame.

Not meaningfully improved?! Four years ago was gpt *3.5*! ChatGPT hadn’t been released!

Re: ARC-AGI Leaderboard

#40

Post by tudelo »

> Only systems which required less than $10,000 to run are shown. (Notes[1])

Am I lost or are their many models on this ranking (Opus 5 included) that clear this?

Post reply on HN