Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arc-AGI-2 and ARC Prize 2025

arcprize.org

Post reply on HN
Print view View original

61–70 of 103 posts

Re: Arc-AGI-2 and ARC Prize 2025

#61

Post by pillefitz »

Chathamization wrote: post #59
Palmik wrote: post #57

Earlier quoted context omitted.

In your example you already indicated two tasks that you think might be hard for AI but easy for humans. Who said that cooking dinner couldn't be part of ARC-AGI- ?

That’s precisely what I meant in my comment by “these types of tests.” People are eventually going to have some sort of standard for what they consider AGI. But that doesn’t mean the current benchmarks are useful for this task at all, and saying that the benchmarks could be completely different in the future only underscores this.

They are useful to reach Arc-N+1

Re: Arc-AGI-2 and ARC Prize 2025

#63

Post by Chathamization »

pillefitz wrote: post #61
Chathamization wrote: post #59

Earlier quoted context omitted.

That’s precisely what I meant in my comment by “these types of tests.” People are eventually going to have some sort of standard for what they consider AGI. But that doesn’t mean the current benchmarks are useful for this task at all, and saying that the benchmarks could be completely different in the future only underscores this.

They are useful to reach Arc-N+1

How are any of these a useful path to asking an AI to cook dinner?

We already know many tasks that most humans can do relatively easily, yet most people don’t expect AI to be able to do them for years to come (for instance, L5 self-driving). ARC-AGI appears to be going in the opposite direction - can these models pass tests that are difficult for the average person to pass.

These benchmarks are interesting in that they show increasing capabilities of the models. But they seem to be far less useful at determining AGI than the simple benchmarks we’ve had all along (can these models do everyday tasks that a human can do?).

Re: Arc-AGI-2 and ARC Prize 2025

#64

Post by az226 »

gkamradt wrote: post #2

Hey HN, Greg from ARC Prize Foundation here. Alongside Mike Knoop and François Francois Chollet, we’re launching ARC-AGI-2, a frontier AI benchmark that measures a model’s ability to generalize on tasks it hasn’t seen before, and the ARC Prize 2025 competition to beat it. In Dec ‘24, ARC-AGI-1 (2019) pinpointed the moment AI moved beyond pure memorization as seen by OpenAI's o3. ARC-AGI-2 targets test-time reasoning.…

Which puzzles had the lowest solve rate? I did the first 10 and felt all easy (mentally solve it in 10-20 seconds for easier ones and 30-60 seconds for harder ones), I’d like to try the most difficult ones.

Re: Arc-AGI-2 and ARC Prize 2025

#65

Post by colordrops »

falcor84 wrote: post #39

I spent half an hour playing with these now at https://arcprize.org/play and it's fun, but I must say that they are not "easy". So far I eventually solved all of the ones I've gone through, but several took me significantly more than the 2 tries allotted. I wonder if this can be shown to be a valid IQ test, and if so, what IQ would a person need to solve e.g. 90% of them in 1 or 2 tries.

Yes, I looked that these and thought about what percentage of humans could even solve these. It seems that, unless average humans are not considered generally intelligence, the test for general intelligence should be passable by most humans.

Re: Arc-AGI-2 and ARC Prize 2025

#66

Post by anshumankmr »

Jensson wrote: post #53
levocardia wrote: post #48

Earlier quoted context omitted.

Ironically "I have a magic AI test but nobody is allowed to use it" is a lot closer to the Yuri Geller situation. Tests are meant to be taken, that should be clear. And...maybe this does not apply in the academic domain, but to some extent if you cheat on an AI test "you're only cheating yourself."

> but to some extent if you cheat on an AI test "you're only cheating yourself." You cheat investors.

And end users and developers and the general public too...

But here is the thing, I feel that even if its rote memorizing why GPT4o couldn't perform just as well on ArcAGI 1 on it or did the "reasoning" help in any way?

Re: Arc-AGI-2 and ARC Prize 2025

#68

Post by [deleted] »

Chathamization wrote: post #55
gkamradt wrote: post #2

Hey HN, Greg from ARC Prize Foundation here. Alongside Mike Knoop and François Francois Chollet, we’re launching ARC-AGI-2, a frontier AI benchmark that measures a model’s ability to generalize on tasks it hasn’t seen before, and the ARC Prize 2025 competition to beat it. In Dec ‘24, ARC-AGI-1 (2019) pinpointed the moment AI moved beyond pure memorization as seen by OpenAI's o3. ARC-AGI-2 targets test-time reasoning.…

> Our belief is that once we can no longer come up with quantifiable problems that are "feasible for humans and hard for AI" then we effectively have AGI. I don’t think that follows. Just because people fail to create ARC-AGI problems that are difficult for an AI to solve, doesn’t mean that said AI can just be plugged into a humanoid robot and it will now reliably cook dinner, order a pizza and drive to pick it up, t…

[deleted]

Re: Arc-AGI-2 and ARC Prize 2025

#69

Post by celdon25 »

gkamradt wrote: post #2

Hey HN, Greg from ARC Prize Foundation here. Alongside Mike Knoop and François Francois Chollet, we’re launching ARC-AGI-2, a frontier AI benchmark that measures a model’s ability to generalize on tasks it hasn’t seen before, and the ARC Prize 2025 competition to beat it. In Dec ‘24, ARC-AGI-1 (2019) pinpointed the moment AI moved beyond pure memorization as seen by OpenAI's o3. ARC-AGI-2 targets test-time reasoning.…

Why wasn’t the ICOM framework (D. Kelley) allowed to make a scoring submission after they claimed to have beaten the scores? Are you concerned that may appear to contradict your mission statement and alienate the AGI community?
Post reply on HN