Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arc Prize 2024 Winners and Technical Report

arcprize.org

Post reply on HN
Print view View original

11–20 of 59 posts

Re: Arc Prize 2024 Winners and Technical Report

#11

Post by trott »

mikeknoop wrote: post #2

Author here -- six months ago we launched ARC Prize, a huge $1M experiment, to test if we need new ideas for AGI. The ARC-AGI benchmark remains unbeaten and I think we can now definitely say "yes". One big update since June is that progress is no longer stalled. Coming into 2024, the public consensus vibe was that pure deep learning / LLMs would continue scaling to AGI. The fundamental architecture of these systems h…

Mike and François,

Compute is limited during inference, and this naturally limits brute-force program search.

But this doesn't prevent one from creating a huge ARC-like dataset ahead of time, like BARC did (but bigger), and training a correspondingly huge NN on it.

Placing a limit on the submission size could foil this kind of brute-force approach though. I wonder if you are considering this for 2025?

Re: Arc Prize 2024 Winners and Technical Report

#13

Post by optimalsolver »

fchollet wrote: post #8
YeGoblynQueenne wrote: post #6

The first question I still have is what happened to core knowledge priors. The white paper that introduced ARC made a big todo about how core knowledge priors are necessary to solve ARC tasks but from what I can tell none of the best-performing (or at-all performing) systems have anything to do with core knowlege priors. So what happened to that assumption? Is it dead? The second question I still have is about the de…

What all top models do is recombine at test time the knowledge they already have. So they all possess Core Knowledge priors. Techniques to acquire them vary: * Use a pretrained LLM and hope that relevant programs will be memorized via exposure to text data (this doesn't work that well) * Pretrain a LLM on ARC-AGI-like data * Hardcode the priors into a DSL > Which is to say, a data augmentation approach The key bit is…

>This demonstrates empirically that ARC-AGI cannot be solved purely via memorization and interpolation

Now that the current challenge is over, and a successor dataset is in the works, can we see how well the leading LLMs perform against the private test set?

Re: Arc Prize 2024 Winners and Technical Report

#14

Post by neoneye2 »

hulium wrote: post #12

Were there any interesting non-neural approaches? I was wondering whether there is any underlying structure in the ARC tasks that could tell us something about algorithms for "reasoning" problems in general.

The 3rd place solution by Agnis Liukis, solves 40 tasks. https://www.kaggle.com/code/gregkamradt/arc-prize-2024-solut...

Re: Arc Prize 2024 Winners and Technical Report

#15

Post by mrandish »

celeritascelery wrote: post #3

What surprises me about this is how poorly general-purpose LLMs do. The best one is OpenAI o1-preview at 18%. This is significantly worse than the purpose-built models like ARChitects (which scored 53.5). This model used TTT to train on the ARC-AGI task specification (amoung other things). It seems that even if someone creates a model that can "solve" ARC, it still is not indicative of AGI since it is not "general" a…

> It seems that even if someone creates a model that can "solve" ARC, it still is not indicative of AGI since it is not "general" anymore

I recently explained why I like ARC to a non-technical friend this way: "When an AI solves ARC it won't be proof of AGI. It's the opposite. As long as ARC remains unsolved I'm confident we're not even close to AGI."

For the sake of being provocative, I'd even argue that ARC remaining unsolved is a sign we're not yet making meaningful progress in the right direction. AGI is the top of Everest. ARC is base camp.

Re: Arc Prize 2024 Winners and Technical Report

#16

Post by aithrowawaycomm »

YeGoblynQueenne wrote: post #6

The first question I still have is what happened to core knowledge priors. The white paper that introduced ARC made a big todo about how core knowledge priors are necessary to solve ARC tasks but from what I can tell none of the best-performing (or at-all performing) systems have anything to do with core knowlege priors. So what happened to that assumption? Is it dead? The second question I still have is about the de…

Even the strongest possible interpretation of the results wouldn't conclude "ARC-AGI is dead" because none of the submissions came especially close to human-level performance; the criteria was 85% success but the best in 2024 was 55%.

That said, I think there should be consideration via information thermodynamics: even with TTT these program-generating systems are using an enormous amount of bits compared to a human mind, a tiny portion of which solves ARC quickly and easily using causality-first principles of reasoning.

Another point: suppose a system solves ARC-AGI with 99% accuracy. Then it should be tested on "HARC-HAGI," a variant that uses hexagons instead of squares. This likely wouldn't trip up a human very much - perhaps a small decrease due to increased surface area for brain farts. But if the AI needs to be retrained on a ton of hexagonal examples, then that AI can't be an AGI candidate.

Re: Arc Prize 2024 Winners and Technical Report

#17

Post by padswo1 »

mikeknoop wrote: post #2

Author here -- six months ago we launched ARC Prize, a huge $1M experiment, to test if we need new ideas for AGI. The ARC-AGI benchmark remains unbeaten and I think we can now definitely say "yes". One big update since June is that progress is no longer stalled. Coming into 2024, the public consensus vibe was that pure deep learning / LLMs would continue scaling to AGI. The fundamental architecture of these systems h…

I don’t think ARC has particularly advanced the research. The approaches that are successful were developed elsewhere and then applied to ARC. Happy to be shown somewhere this is not the case.

In the case of TTT, I wouldn’t really describe that as a ‘new AGI reasoning approach’. People have been fine tuning deep learning models on specific tasks for a long time.

The fundamental instinct driving the creation of ARC - that ‘deep learning cannot do system 2 thinking’, is under threat of being proven wrong very soon. Attempts to define the approaches that are working as somehow not ‘traditional deep learning’ really seem like shifting the goal posts.

Re: Arc Prize 2024 Winners and Technical Report

#18

Post by mikeknoop »

tbalsam wrote: post #10
mikeknoop wrote: post #2

Author here -- six months ago we launched ARC Prize, a huge $1M experiment, to test if we need new ideas for AGI. The ARC-AGI benchmark remains unbeaten and I think we can now definitely say "yes". One big update since June is that progress is no longer stalled. Coming into 2024, the public consensus vibe was that pure deep learning / LLMs would continue scaling to AGI. The fundamental architecture of these systems h…

As a rather experienced ML researcher, ARC is a great benchmark on its own, but is punching below its weight in terms of claiming that it is a gate (or in terms of this post -- a "steward") towards AGI, and in my perspective and the perspective of several researchers near me this has watered down the value of the ARC benchmark as a test. It is a great unit test for reasoning -- that's fantastic! And maybe it is indee…

> I'd heartily recommend maybe taking down the marketing vibrance down a notch and keep things a bit more measured, it's not entirely a meme, though some of the more-serious researchers don't take it as seriously as a result.

This is fair critique. ARC Prize's 2024 messaging was sharp to break through the noise floor -- ARC has been around since 2019 but most only learned about it this summer. Now that it has garnered awareness, it is no longer useful, and in same cases hurting progress like you point out. The messaging needs to evolve and mature next year to be more neutral/academic.

Re: Arc Prize 2024 Winners and Technical Report

#19

Post by YeGoblynQueenne »

fchollet wrote: post #8
YeGoblynQueenne wrote: post #6

The first question I still have is what happened to core knowledge priors. The white paper that introduced ARC made a big todo about how core knowledge priors are necessary to solve ARC tasks but from what I can tell none of the best-performing (or at-all performing) systems have anything to do with core knowlege priors. So what happened to that assumption? Is it dead? The second question I still have is about the de…

What all top models do is recombine at test time the knowledge they already have. So they all possess Core Knowledge priors. Techniques to acquire them vary: * Use a pretrained LLM and hope that relevant programs will be memorized via exposure to text data (this doesn't work that well) * Pretrain a LLM on ARC-AGI-like data * Hardcode the priors into a DSL > Which is to say, a data augmentation approach The key bit is…

>> So they all possess Core Knowledge priors.

Do you mean the ones from your white paper? The same ones that humans possess? How do you know this?

>> The key bit isn't the data augmentation but the TTT.

I haven't had the chance to read the papers carefully. Have they done ablation studies? For instance, is the following a guess or is it an empirical result?

>> For instance, if you drop the TTT component you will see that these large models trained on millions of synthetic ARC-AGI tasks drop to <10% accuracy.

Re: Arc Prize 2024 Winners and Technical Report

#20

Post by tbalsam »

mikeknoop wrote: post #18
tbalsam wrote: post #10

Earlier quoted context omitted.

As a rather experienced ML researcher, ARC is a great benchmark on its own, but is punching below its weight in terms of claiming that it is a gate (or in terms of this post -- a "steward") towards AGI, and in my perspective and the perspective of several researchers near me this has watered down the value of the ARC benchmark as a test. It is a great unit test for reasoning -- that's fantastic! And maybe it is indee…

> I'd heartily recommend maybe taking down the marketing vibrance down a notch and keep things a bit more measured, it's not entirely a meme, though some of the more-serious researchers don't take it as seriously as a result. This is fair critique. ARC Prize's 2024 messaging was sharp to break through the noise floor -- ARC has been around since 2019 but most only learned about it this summer. Now that it has garnere…

I feel rather consternated that this response effectively boils down to "yes, we know we overhyped this to get people's attention, and now that we have it we can be more honest about it". Fighting for place in the attention economy is understandable, being deceptive about it is not.

This is part of the ethical morass of why some more serious researchers aren't touching the benchmark. People are not going to take it seriously if it continues like this!

Post reply on HN