Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Day 1 of ARC-AGI-3

symbolica.ai

Post reply on HN
Print view View original

71–79 of 79 posts

Re: Day 1 of ARC-AGI-3

#71

Post by modeless »

andy12_ wrote: post #53

Apparently the score would be a little higher if it weren't for the fact that scores are penalized for being worse than the human baseline, but aren't rewarded for being better than the human baseline (which seems like an arbitrary decision. The human baseline is not optimal).

Once you have matched humans on a problem then further progress on that problem is not necessarily meaningful anymore, in terms of quantitative measurement of intelligence. ARC-AGI-3 is designed to compare AIs to humans, not to measure arbitrarily high levels of superhuman intelligence. For that you would want a different benchmark.

Re: Day 1 of ARC-AGI-3

#72

Post by fc417fc802 »

UltraSane wrote: post #67
fc417fc802 wrote: post #61

Earlier quoted context omitted.

Rephrase that in terms of the human mechanic and hopefully you can see the error of that reasoning. LLMs that perform tasks (as opposed to merely holding conversations) use tools just like we do. That's literally how we design them to operate. In fact the LLMs that everyone uses today typically have access to specialized task specific tooling. Obviously specialized tools aren't appropriate for a test that measures th…

I'm pretty sure the LLM can use tools while doing arc-agi-3 but it has to the same tools available all the time not an incredibly elaborate custom harness.

To quote someone else from upthread, tool use requires a harness. Without one an LLM as commonly understood is a bare model that receives inputs and directly produces outputs the same as talking to an unaided person.

Re: Day 1 of ARC-AGI-3

#73

Post by cxdorn »

DetroitThrow wrote: post #68
cxdorn wrote: post #66

Earlier quoted context omitted.

(disclaimer: i worked on early versions of agentica_sdk; but wasn't involved in recent developments and the ARC solver) As other comments point out this is about harness development and harness efficiency. Agentica SDK is a sort of meta harness, that makes things easy: plug any "internal API" (as defined natively in your codebase) directly into your agent. Agentica SDK itself is not application specifc; but the APIs…

>In the world of harness development I think that's an interesting question to answer! The challenge isn't about harness development though, and a sufficiently complex harness can solve these tasks rather easily. And presenting it as if you've made a novel development for solving ARC-AGI-3 leads me to believe you're willing to waste all of our time for your benefit at every step in the future.

> a sufficiently complex harness can solve these tasks rather easily.

I claim this is not so easily done, and earlier iterations of ARC-AGI did not have the constraint in the first place. You want something that generalizes across all puzzles (hopefully even the private ones), and these puzzles are extremely diverse ... and hard; telling the model the controls and some basic guidelines for the game is the only "obvious" thing you can do.

The other point of my reply was efficiency, both in terms of creating and using the harness; the discussed solution is something that anyone (in fact, likely even an LLM itself) can cook up in a few minutes; it's not much more than a game control wrapper so the agent can play around with the game in live python and some generalities as laid out in the prompt.

(But I'm always happy to be proven wrong. What harnesses did you have in mind?)

Re: Day 1 of ARC-AGI-3

#74

Post by falcor84 »

sanxiyn wrote: post #7
falcor84 wrote: post #6

Earlier quoted context omitted.

I went through the technical paper again, and while they explain why they decided against the harness, I disagree with them - my take is that if harnesses are overfitting, then they should be penalized on the hidden test set. Anyway, searching both in ARC-AGI's paper and website and directly on kaggle, I failed to find a with-harness leaderboard; can you please give the link?

Here it is: https://arcprize.org/leaderboard/community

Ah, it's based on this repo [0] and there's only 1 non-example submission there [1], from 2 weeks ago (so it only covers the preview games), and their schema doesn't a field to show that it's only the preview, nor does the thing properly parse the score or cost into the table. And the biggest thing is that apparently there's no validation whatsoever - submissions are not ever run on the hidden test games, so is essentially useless as a comparison.

[0] https://github.com/arcprize/ARC-AGI-Community-Leaderboard [1] https://github.com/arcprize/ARC-AGI-Community-Leaderboard/bl...

Re: Day 1 of ARC-AGI-3

#75

Post by UltraSane »

fc417fc802 wrote: post #72
UltraSane wrote: post #67

Earlier quoted context omitted.

I'm pretty sure the LLM can use tools while doing arc-agi-3 but it has to the same tools available all the time not an incredibly elaborate custom harness.

To quote someone else from upthread, tool use requires a harness. Without one an LLM as commonly understood is a bare model that receives inputs and directly produces outputs the same as talking to an unaided person.

Then the LLM has to write the harness.

Re: Day 1 of ARC-AGI-3

#76

Post by fc417fc802 »

UltraSane wrote: post #75
fc417fc802 wrote: post #72

Earlier quoted context omitted.

To quote someone else from upthread, tool use requires a harness. Without one an LLM as commonly understood is a bare model that receives inputs and directly produces outputs the same as talking to an unaided person.

Then the LLM has to write the harness.

I'd like to suggest that prior to expressing disagreement you really ought to reread the comment you're replying to and make sure your understanding is correct.

Quoting this for the second time now - tool use requires a harness.

Without a harness the LLM has no ability to interact with the world. It has no agency. It's just spitting out text (or whatever else) into the void. There's no programming tools, no filesystem, no shell, nothing.

Re: Day 1 of ARC-AGI-3

#77

Post by UltraSane »

fc417fc802 wrote: post #76
UltraSane wrote: post #75

Earlier quoted context omitted.

Then the LLM has to write the harness.

I'd like to suggest that prior to expressing disagreement you really ought to reread the comment you're replying to and make sure your understanding is correct. Quoting this for the second time now - tool use requires a harness. Without a harness the LLM has no ability to interact with the world. It has no agency. It's just spitting out text (or whatever else) into the void. There's no programming tools, no filesyste…

And by the rules of arc-agi-3 the LLM will have to write any harness it needs. I'm not sure what we are even arguing about this point.

Re: Day 1 of ARC-AGI-3

#79

Post by boxed »

nubg wrote: post #40
bytesandbits wrote: post #34

we constantly underestimate the power of inference scaffolding. I have seen it in all domains: coding, ASR, ARC-AGI benchmarks you name it. Scaffolding can do a lot! And post-training too. I am confident our currently pre-trained models can beat this benchmark over 80% with the right post-training and scaffolding. That being said I don't think ARC-AGI proves much. It is not a useful task at all in the wild. it is jus…

what exactly does scaffolding mean in this context? genuine question

I'm gonna guess it means "whatever we still need humans to figure out to spoon feed the models"
Post reply on HN