Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arc-AGI-2 and ARC Prize 2025

arcprize.org

Post reply on HN
Print view View original

91–100 of 103 posts

Re: Arc-AGI-2 and ARC Prize 2025

#91

Post by torginus »

littlestymaar wrote: post #72
synapsomorphy wrote: post #15

Earlier quoted context omitted.

Thanks for your awesome work Greg! The success of o3 directly contradicts us being in an "idea-constrained environment", what makes you believe that?

What makes you think so? From ChatGPT 3.5 to o1, all LLMs progress came from investment in training: either by using much more data, or using higher quality data thanks to artificial data. o1 (and then o3) broke this paradigm by applying a novel idea (RL+search on CoT) and that's because of it that it was able to make progress on ARC-AGI. So IMO the success of o3 goes in favor of the argument of how we are in an idea…

This isn't a novel idea - some people tried the exact same thing the day GPT4 came out.

And going back even further, there's Goal Oriented Action Planning - an old timey video game AI technique, that's basically searching through solution space to construct a plan:

https://medium.com/@vedantchaudhari/goal-oriented-action-pla...

(besides the fact that almost all old timey AI is state space solution search)

Re: Arc-AGI-2 and ARC Prize 2025

#92

Post by CooCooCaCha »

Chathamization wrote: post #55
gkamradt wrote: post #2

Hey HN, Greg from ARC Prize Foundation here. Alongside Mike Knoop and François Francois Chollet, we’re launching ARC-AGI-2, a frontier AI benchmark that measures a model’s ability to generalize on tasks it hasn’t seen before, and the ARC Prize 2025 competition to beat it. In Dec ‘24, ARC-AGI-1 (2019) pinpointed the moment AI moved beyond pure memorization as seen by OpenAI's o3. ARC-AGI-2 targets test-time reasoning.…

> Our belief is that once we can no longer come up with quantifiable problems that are "feasible for humans and hard for AI" then we effectively have AGI. I don’t think that follows. Just because people fail to create ARC-AGI problems that are difficult for an AI to solve, doesn’t mean that said AI can just be plugged into a humanoid robot and it will now reliably cook dinner, order a pizza and drive to pick it up, t…

The statement you quoted is a general statement, not specific to ARC-AGI.

The scenarios you listed are examples of what they’re talking about. Those are tasks that humans can easily do but robots have a hard time with.

Re: Arc-AGI-2 and ARC Prize 2025

#93

Post by YeGoblynQueenne »

szvsw wrote: post #37
YeGoblynQueenne wrote: post #33

Earlier quoted context omitted.

>> The reason these tasks require fluid intelligence is because they were designed this way -- with task uniqueness/novelty as the primary goal. That's in no way different than claiming that LLMs understand language, or reason, etc, because they were designed that way. Neural nets of all sorts have been beating benchmarks since forever, e.g. there's a ton of language understanding benchmarks pretty much all saturated…

I mostly agree with what your are saying but… > passing them means nothing about the ability of neural net-based systems to understand language, regardless of how much their authors designed them to test language understanding. Does this implicitly suggest that it is impossible to quantitatively assess a system’s ability to understand language? (Using the term “system” in the broadest possible sense) Not agreeing or…

>> Does this implicitly suggest that it is impossible to quantitatively assess a system’s ability to understand language? (Using the term “system” in the broadest possible sense)

I don't know and I don't have an opinion. I know that tests that claimed to measure language understanding, historically, haven't. There's some literature on the subject if you're curious (sounds like you are). I'd start here:

Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data

Emily M. Bender, Alexander Koller

https://aclanthology.org/2020.acl-main.463/

Quoting the passage that I tend to remember:

>> While large neural LMs may well end up being important components of an eventual full-scale solution to human-analogous NLU, they are not nearly-there solutions to this grand challenge. We argue in this paper that genuine progress in our field — climbing the right hill, not just the hill on whose slope we currently sit —depends on maintaining clarity around big picture notions such as meaning and understanding in task design and reporting of experimental results.

Re: Arc-AGI-2 and ARC Prize 2025

#94

Post by YeGoblynQueenne »

fchollet wrote: post #36
YeGoblynQueenne wrote: post #33

Earlier quoted context omitted.

>> The reason these tasks require fluid intelligence is because they were designed this way -- with task uniqueness/novelty as the primary goal. That's in no way different than claiming that LLMs understand language, or reason, etc, because they were designed that way. Neural nets of all sorts have been beating benchmarks since forever, e.g. there's a ton of language understanding benchmarks pretty much all saturated…

The first time a top lab spent millions trying to beat ARC was actually in 2021, and the effort failed. By the time OpenAI attempted ARC in 2024, a colossal amount of resources had already been expended trying to beat the benchmark. The OpenAI run itself costs several millions in inference compute alone. ARC was the only benchmark that highlighted o3 as having qualitatively different abilities compared to all models…

>> The first time a top lab spent millions trying to beat ARC was actually in 2021, and the effort failed.

Which top lab was that? What did they try?

>> ARC was the only benchmark that highlighted o3 as having qualitatively different abilities compared to all models that came before.

Unfortunately observations support a simpler hypothesis: o3 was trained on sufficient data about ARC-1 that it could solve it well. There is currently insufficient data on ARC-II to solve it therefore o3 can't solve it. No super magickal and mysterious qualitatively different abilities to all models that came before required whatsoever.

Indeed, that is a common pattern in machine learning research: newer models perform better on benchmarks than earlier models not because their capabilities increase with respect to earlier models but because they're bigger models, trained on more data and more compute. They're just bigger, slower, more expensive- and just as dumb as their predecessors.

That's 90% of deep learning research in a nutshell.

Re: Arc-AGI-2 and ARC Prize 2025

#95

Post by mchusma »

Chathamization wrote: post #63
pillefitz wrote: post #61

Earlier quoted context omitted.

They are useful to reach Arc-N+1

How are any of these a useful path to asking an AI to cook dinner? We already know many tasks that most humans can do relatively easily, yet most people don’t expect AI to be able to do them for years to come (for instance, L5 self-driving). ARC-AGI appears to be going in the opposite direction - can these models pass tests that are difficult for the average person to pass. These benchmarks are interesting in that th…

Genuine question, do you feel Waymo is not L5 self-driving? I Waymo has L5 but its not truly economic yet.

Re: Arc-AGI-2 and ARC Prize 2025

#96

Post by bubblyworld »

YeGoblynQueenne wrote: post #94
fchollet wrote: post #36

Earlier quoted context omitted.

The first time a top lab spent millions trying to beat ARC was actually in 2021, and the effort failed. By the time OpenAI attempted ARC in 2024, a colossal amount of resources had already been expended trying to beat the benchmark. The OpenAI run itself costs several millions in inference compute alone. ARC was the only benchmark that highlighted o3 as having qualitatively different abilities compared to all models…

>> The first time a top lab spent millions trying to beat ARC was actually in 2021, and the effort failed. Which top lab was that? What did they try? >> ARC was the only benchmark that highlighted o3 as having qualitatively different abilities compared to all models that came before. Unfortunately observations support a simpler hypothesis: o3 was trained on sufficient data about ARC-1 that it could solve it well. The…

I'm sorry, but what observations support that hypothesis? There were scores of teams trying exactly that - training LLMs directly on Arc-AGI data - and by and large they achieved mediocre results. It just isn't an approach that works for this problem set.

To be honest your argument sounds like an attempt to motivate a predetermined conclusion.

Re: Arc-AGI-2 and ARC Prize 2025

#97

Post by littlestymaar »

torginus wrote: post #91
littlestymaar wrote: post #72

Earlier quoted context omitted.

What makes you think so? From ChatGPT 3.5 to o1, all LLMs progress came from investment in training: either by using much more data, or using higher quality data thanks to artificial data. o1 (and then o3) broke this paradigm by applying a novel idea (RL+search on CoT) and that's because of it that it was able to make progress on ARC-AGI. So IMO the success of o3 goes in favor of the argument of how we are in an idea…

This isn't a novel idea - some people tried the exact same thing the day GPT4 came out. And going back even further, there's Goal Oriented Action Planning - an old timey video game AI technique, that's basically searching through solution space to construct a plan: https://medium.com/@vedantchaudhari/goal-oriented-action-pla... (besides the fact that almost all old timey AI is state space solution search)

What's new is to apply that to LLMs, that is.

> This isn't a novel idea - some people tried the exact same thing the day GPT4 came out.

What do you mean? Since GPT4's weights aren't available, you can't run RL on it by yourself. Only OpenAI can.

Re: Arc-AGI-2 and ARC Prize 2025

#98

Post by cubefox »

colordrops wrote: post #65
falcor84 wrote: post #39

I spent half an hour playing with these now at https://arcprize.org/play and it's fun, but I must say that they are not "easy". So far I eventually solved all of the ones I've gone through, but several took me significantly more than the 2 tries allotted. I wonder if this can be shown to be a valid IQ test, and if so, what IQ would a person need to solve e.g. 90% of them in 1 or 2 tries.

Yes, I looked that these and thought about what percentage of humans could even solve these. It seems that, unless average humans are not considered generally intelligence, the test for general intelligence should be passable by most humans.

I would argue that also small children and even most animals count as "general" intelligences. Animals are much less intelligent than grown humans, but that doesn't mean they are less general. Just like, say, AlphaGo 2 is more intelligent but not more general than AlphaGo 1. Or Qwen 32B vs Qwen 7B. Model or brain size alone doesn't determine generality. Generality is more a question of architecture.

Re: Arc-AGI-2 and ARC Prize 2025

#99

Post by colordrops »

cubefox wrote: post #98
colordrops wrote: post #65

Earlier quoted context omitted.

Yes, I looked that these and thought about what percentage of humans could even solve these. It seems that, unless average humans are not considered generally intelligence, the test for general intelligence should be passable by most humans.

I would argue that also small children and even most animals count as "general" intelligences. Animals are much less intelligent than grown humans, but that doesn't mean they are less general. Just like, say, AlphaGo 2 is more intelligent but not more general than AlphaGo 1. Or Qwen 32B vs Qwen 7B. Model or brain size alone doesn't determine generality. Generality is more a question of architecture.

Is there a formal or at least clear consensus definition of "general" intelligence? I assume it involves some level of autonomy and ability to manage novel situations.

Re: Arc-AGI-2 and ARC Prize 2025

#100

Post by jwpapi »

fchollet wrote: post #34
jwpapi wrote: post #32

Did we run out of textual tasks that are easy for humans but hard for AI, or why are the examples all graphics?

You can easily convert these tasks to token strings. The reason why ARC does not use language as part of its format is that it seeks to minimize the amount of prior knowledge needed to approach the tasks, so as to focus on fluid intelligence as opposed to acquired knowledge. All ARC tasks are built entirely on top of "Core Knowledge" priors, the kind of elementary knowledge that a small child has already mastered and…

Can you explain to me? Would the token strings be as easy to solve for humans as well?

Or let me ask differently. Can we still design text questions that are easy for humans and tough for AI?

Post reply on HN