Earlier quoted context omitted.
Hey, he's the visionary. You come up with the nuts and bolts.
is keras nuts and bolts enough?
Keres is a good abstraction model but poorly implemented.
Arc Prize 2024 Winners and Technical Report
51–59 of 59 posts
Re: Arc Prize 2024 Winners and Technical Report
#52What surprises me about this is how poorly general-purpose LLMs do. The best one is OpenAI o1-preview at 18%. This is significantly worse than the purpose-built models like ARChitects (which scored 53.5). This model used TTT to train on the ARC-AGI task specification (amoung other things). It seems that even if someone creates a model that can "solve" ARC, it still is not indicative of AGI since it is not "general" a…
It is correct that the first model that will beat ARC-AGI will only be able to handle ARC-AGI tasks. However, the idea is that the architecture of that model should be able to be repurposed to arbitrary problems. That is what makes ARC-AGI a good compass towards AGI (unlike chess). For instance, current top models use TTT, which is a completely general-purpose technique that provides the most significant boost to DL…
"However, the idea is that the architecture of that model should be able to be repurposed to arbitrary problems"
From a mathematical perspective, this doesn't sound right. All NNs are universal apprxomators and in theory can all learn the same thing to equal ability. It's more about the learning algorithm than the architecture IMO.
Re: Arc Prize 2024 Winners and Technical Report
#53Were there any interesting non-neural approaches? I was wondering whether there is any underlying structure in the ARC tasks that could tell us something about algorithms for "reasoning" problems in general.
This is a good question!!
Re: Arc Prize 2024 Winners and Technical Report
#54Were there any interesting non-neural approaches? I was wondering whether there is any underlying structure in the ARC tasks that could tell us something about algorithms for "reasoning" problems in general.
The 3rd place solution by Agnis Liukis, solves 40 tasks. https://www.kaggle.com/code/gregkamradt/arc-prize-2024-solut...
Can we get this person to explain thier inspiration?
Re: Arc Prize 2024 Winners and Technical Report
#55Earlier quoted context omitted.
>> Now that the AI research field is coming around to the idea that something beyond deep learning is needed, the story matters less, and the benchmark, and future versions, can stand on their utility as a compass towards AGI. How so? All the three top systems are deep neural net systems. The first place went to a system that, quoting from the "contributions" section of the paper, employed: >> An automated data gener…
> realise that you're wrong, accept it, and move on What do you think about limiting the submission size? Kaggle does this sometimes. With a limit like 0.1-1MB (compressed), you are basically saying: "Give me sample-efficient learning algorithms, not pretrained models."
That's fine if you want to measure sample efficiency, but ARC-AGI is supposed to measure progress towards AGI.
Re: Arc Prize 2024 Winners and Technical Report
#56Earlier quoted context omitted.
> realise that you're wrong, accept it, and move on What do you think about limiting the submission size? Kaggle does this sometimes. With a limit like 0.1-1MB (compressed), you are basically saying: "Give me sample-efficient learning algorithms, not pretrained models."
That's fine if you want to measure sample efficiency, but ARC-AGI is supposed to measure progress towards AGI.
> That's fine if you want to measure sample efficiency, but ARC-AGI is supposed to measure progress towards AGI.
On the Measure of Intelligence defines intelligence as skill-acquisition efficiency, I believe, where efficiency is with respect to whatever is the limiting factor. For each ARC task, the primary limiting factor is the number of samples in it. And the skill here is your ability to convert inputs into the correct outputs. In other words, in this context, intelligence is sample-efficiency, as I see it.
Re: Arc Prize 2024 Winners and Technical Report
#57Author here -- six months ago we launched ARC Prize, a huge $1M experiment, to test if we need new ideas for AGI. The ARC-AGI benchmark remains unbeaten and I think we can now definitely say "yes". One big update since June is that progress is no longer stalled. Coming into 2024, the public consensus vibe was that pure deep learning / LLMs would continue scaling to AGI. The fundamental architecture of these systems h…
> Coming into 2024, the public consensus vibe was that pure deep learning / LLMs would continue scaling to AGI.
Was it? What did the "public" consist of exactly?
Re: Arc Prize 2024 Winners and Technical Report
#58Earlier quoted context omitted.
That's fine if you want to measure sample efficiency, but ARC-AGI is supposed to measure progress towards AGI.
> That's fine if you want to measure sample efficiency, but ARC-AGI is supposed to measure progress towards AGI. On the Measure of Intelligence defines intelligence as skill-acquisition efficiency, I believe, where efficiency is with respect to whatever is the limiting factor. For each ARC task, the primary limiting factor is the number of samples in it. And the skill here is your ability to convert inputs into the c…
Is that what fchollet is claiming?
Re: Arc Prize 2024 Winners and Technical Report
#59Earlier quoted context omitted.
> That's fine if you want to measure sample efficiency, but ARC-AGI is supposed to measure progress towards AGI. On the Measure of Intelligence defines intelligence as skill-acquisition efficiency, I believe, where efficiency is with respect to whatever is the limiting factor. For each ARC task, the primary limiting factor is the number of samples in it. And the skill here is your ability to convert inputs into the c…
Is that what fchollet is claiming?
Not sure. But I think this follows logically from the definition of intelligence he is using. Also, see II.2.2 in the paper.