Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Day 1 of ARC-AGI-3

symbolica.ai

Post reply on HN
Print view View original

61–70 of 79 posts

Re: Day 1 of ARC-AGI-3

#61

Post by fc417fc802 »

UltraSane wrote: post #60
fc417fc802 wrote: post #59

Earlier quoted context omitted.

No, that would be analogous to disallowing customized harnesses, ie tooling specially crafted by someone else for the specific task at hand. Insisting that an LLM solve something without the ability to make use of any external tooling whatsoever is almost perfectly analogous to insisting that a human mechanic work on a car with nothing but his own bare hands. The wrench is to the mechanic as the stock python repl is…

They want the LLM that does the ARC-AGI-3 to be the same LLM that everyone uses.

Rephrase that in terms of the human mechanic and hopefully you can see the error of that reasoning. LLMs that perform tasks (as opposed to merely holding conversations) use tools just like we do. That's literally how we design them to operate.

In fact the LLMs that everyone uses today typically have access to specialized task specific tooling. Obviously specialized tools aren't appropriate for a test that measures the ability to generalize but generic tools are par for the course. Writing a bot to play a game for you would certainly serve to demonstrate an understanding of the task.

Re: Day 1 of ARC-AGI-3

#62

Post by versteegen »

stephantul wrote: post #39

The fact that this was on the set of training problems with a custom harness basically makes the headline a lie. What if you give opus the same harness? Do people even care about meaningful comparisons any more or is it all just “numbers go up”

...Their agent is called "Agentica ARC-AGI-3 agent for Opus 4.6 (120k) High".

Yes, it's unfair to compare results for the 25 (easier) public games against scores for the 55 semi-private games (scores for which are taken from https://arcprize.org/leaderboard).

But you're wrong to say that a custom harness invalidates the result. Yes, the official "ARC verified" scoreboard for frontier LLMs requires (https://arcprize.org/policy):

> using extremely generic and miminal LLM testing prompts, no client-side "harnesses", no hand-crafted tools, and no tailored model configuration

but these are limitations placed in order to compare LLMs from frontier labs on equal footing, not limitations that apply to submissions in general. It's not as if a solution to ARC-AGI-3 must involve training a custom LLM! This Agentica harness is completely legitimate approach to ARC-AGI-3, similar to J. Berman's for ARC-AGI-1/2, for example.

Re: Day 1 of ARC-AGI-3

#63

Post by versteegen »

mohsen1 wrote: post #47

Uses public dataset to evaluate which is not meant for evaluation. Writes super specific prompt[1] and claims eye catching results. This is the state of "AI" these days I guess... [1] https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...

The dataset miscomparison is a big problem. The prompt is super specific to ARC-AGI-3, which is perfectly fine to do, but skimming it I saw nothing that appears specific to the 25 games in the dataset. Especially considering they've only had one day for overfitting. Could be quite subtle leakage though.

Re: Day 1 of ARC-AGI-3

#64

Post by versteegen »

bberrry wrote: post #52
sanxiyn wrote: post #14

Earlier quoted context omitted.

In this case the code is public and you can see they are not cheating in that sense.

They are definitely cheating, they have crafted prompts[1] that explain the game rules rather than have the model explore and learn. 1. https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...

Where do you see that? I only skimmed the prompts but don't see any aspects of any of the games explained in there. There are a few hints which are legitimate prior knowledge about games in general, though some looks too inflexible to me. Prior knowledge ("Core priors") is a critical requirement of the ARC series, read the reports.

Re: Day 1 of ARC-AGI-3

#65

Post by GorbachevyChase »

padolsey wrote: post #42

Knowing the nature of a test ahead of time, building out your capabilities and tooling before entering the exam hall when your peers don't have that advantage, makes you a cheater.

Humans who have played games should also not be allowed to test in ARC AGI. Cavemen only.

Re: Day 1 of ARC-AGI-3

#66

Post by cxdorn »

fchollet wrote: post #20
lairv wrote: post #2

Note that this uses a harness so it doesn't qualify for the official ARC-AGI-3 leaderboard According to the authors the harness isn't ARC-AGI specific though https://x.com/agenticasdk/status/2037335806264971461

It is 100% ARC-AGI-3 specific though, just read through the prompts https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...

(disclaimer: i worked on early versions of agentica_sdk; but wasn't involved in recent developments and the ARC solver)

As other comments point out this is about harness development and harness efficiency. Agentica SDK is a sort of meta harness, that makes things easy: plug any "internal API" (as defined natively in your codebase) directly into your agent. Agentica SDK itself is not application specifc; but the APIs of your application are... application specific.

Re: the linked prompt. A harness is a set of tools and descriptions how to best use those tools, and sometimes some external control flow based on the outcome of using those tools. How to "best use the tools" should always be part of the prompt (like in this case).

So this work tries to answer: "short of telling the agent any solutions, make a simple but efficient API to play the games, hand it to the agent, and see how it does". In the world of harness development I think that's an interesting question to answer!

Re: Day 1 of ARC-AGI-3

#67

Post by UltraSane »

fc417fc802 wrote: post #61
UltraSane wrote: post #60

Earlier quoted context omitted.

They want the LLM that does the ARC-AGI-3 to be the same LLM that everyone uses.

Rephrase that in terms of the human mechanic and hopefully you can see the error of that reasoning. LLMs that perform tasks (as opposed to merely holding conversations) use tools just like we do. That's literally how we design them to operate. In fact the LLMs that everyone uses today typically have access to specialized task specific tooling. Obviously specialized tools aren't appropriate for a test that measures th…

I'm pretty sure the LLM can use tools while doing arc-agi-3 but it has to the same tools available all the time not an incredibly elaborate custom harness.

Re: Day 1 of ARC-AGI-3

#68

Post by DetroitThrow »

cxdorn wrote: post #66
fchollet wrote: post #20

Earlier quoted context omitted.

It is 100% ARC-AGI-3 specific though, just read through the prompts https://github.com/symbolica-ai/ARC-AGI-3-Agents/blob/symbol...

(disclaimer: i worked on early versions of agentica_sdk; but wasn't involved in recent developments and the ARC solver) As other comments point out this is about harness development and harness efficiency. Agentica SDK is a sort of meta harness, that makes things easy: plug any "internal API" (as defined natively in your codebase) directly into your agent. Agentica SDK itself is not application specifc; but the APIs…

>In the world of harness development I think that's an interesting question to answer!

The challenge isn't about harness development though, and a sufficiently complex harness can solve these tasks rather easily.

And presenting it as if you've made a novel development for solving ARC-AGI-3 leads me to believe you're willing to waste all of our time for your benefit at every step in the future.

Re: Day 1 of ARC-AGI-3

#69

Post by stephantul »

ting0 wrote: post #45
stephantul wrote: post #39

The fact that this was on the set of training problems with a custom harness basically makes the headline a lie. What if you give opus the same harness? Do people even care about meaningful comparisons any more or is it all just “numbers go up”

Does it matter though? If it accomplishes the task, it accomplishes the task. Everyone uses a harness anyway, and finding the best harness is relevant. Also perhaps this hints at something bigger, i.e.: we're wasting our time focusing on the model when we could be focusing on the harness.

Yes it matters, because it’s not a measurement of whether it accomplishes the task if a human tells it how to solve it.

Re: Day 1 of ARC-AGI-3

#70

Post by stephantul »

versteegen wrote: post #62
stephantul wrote: post #39

The fact that this was on the set of training problems with a custom harness basically makes the headline a lie. What if you give opus the same harness? Do people even care about meaningful comparisons any more or is it all just “numbers go up”

...Their agent is called "Agentica ARC-AGI-3 agent for Opus 4.6 (120k) High". Yes, it's unfair to compare results for the 25 (easier) public games against scores for the 55 semi-private games (scores for which are taken from https://arcprize.org/leaderboard ). But you're wrong to say that a custom harness invalidates the result. Yes, the official "ARC verified" scoreboard for frontier LLMs requires ( https://arcprize…

I’m not saying it invalidates the result. I am saying that they knew the headline and comparison was not correct and they still decided to roll with it. It’s an incorrect representation of what happened, designed to get eyeballs and possibly vc dollars.
Post reply on HN