Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arc-AGI-2 and ARC Prize 2025

arcprize.org

Post reply on HN
Print view View original

11–20 of 103 posts

Re: Arc-AGI-2 and ARC Prize 2025

#11

Post by zamadatix »

gkamradt wrote: post #4
artninja1988 wrote: post #3

Earlier quoted context omitted.

What are you doing to prevent the test set being leaked? Will you still be offering API access to the semi private test set to the big model providers who presumably train on their API?

We have a few sets: 1. Public Train - 1,000 tasks that are public 2. Public Eval - 120 tasks that are public So for those two we don't have protections. 3. Semi Private Eval - 120 tasks that are exposed to 3rd parties. We sign data agreements where we can, but we understand this is exposed and not 100% secure. It's a risk we are open to in order to keep testing velocity. In theory it is very difficulty to secure this…

What prevents everything in 4 from becoming a part of 3 the first time the test set is run on a proprietary model, do you require competitors like OpenAI provide models Kaggle can self host for the test?

Re: Arc-AGI-2 and ARC Prize 2025

#12

Post by gkamradt »

FergusArgyll wrote: post #9

I'd love to hear from the ARC guys: These benchmarks, and specifically the constraints placed on solving them (compute etc) seem to me to incentivize the opposite of "general intelligence" Have any of the technical contributions used to win the past competition been used to advance general AI in any way? We have transformer based systems constantly gaining capabilities. On the other hand have any of the Kaggle submis…

Good question! This was one of the main motivations of our "Paper Prize" track. We wanted to reward conceptual progress vs leaderboard chasing. In fact, when we increased the prizes mid year we awarded more money towards the paper track vs top score.

We had 40 papers submitted last year and 8 were awarded prizes. [1]

On of the main teams, MindsAI, just published their paper on their novel test time fine tuning approach. [2]

Jan/Daniel (1st place winners last year) talk all about their progress and journey building out here [3]. Stories like theirs help push the field forward.

[1] https://arcprize.org/blog/arc-prize-2024-winners-technical-r...

[2] https://github.com/MohamedOsman1998/deep-learning-for-arc/bl...

[3] https://www.youtube.com/watch?v=mTX_sAq--zY

Re: Arc-AGI-2 and ARC Prize 2025

#13

Post by gkamradt »

zamadatix wrote: post #11
gkamradt wrote: post #4

Earlier quoted context omitted.

We have a few sets: 1. Public Train - 1,000 tasks that are public 2. Public Eval - 120 tasks that are public So for those two we don't have protections. 3. Semi Private Eval - 120 tasks that are exposed to 3rd parties. We sign data agreements where we can, but we understand this is exposed and not 100% secure. It's a risk we are open to in order to keep testing velocity. In theory it is very difficulty to secure this…

What prevents everything in 4 from becoming a part of 3 the first time the test set is run on a proprietary model, do you require competitors like OpenAI provide models Kaggle can self host for the test?

#4 (private test set) doesn't get used for any public model testing. It is only used on the Kaggle leaderboard where no internet access is allowed.

Re: Arc-AGI-2 and ARC Prize 2025

#15

Post by synapsomorphy »

gkamradt wrote: post #2

Hey HN, Greg from ARC Prize Foundation here. Alongside Mike Knoop and François Francois Chollet, we’re launching ARC-AGI-2, a frontier AI benchmark that measures a model’s ability to generalize on tasks it hasn’t seen before, and the ARC Prize 2025 competition to beat it. In Dec ‘24, ARC-AGI-1 (2019) pinpointed the moment AI moved beyond pure memorization as seen by OpenAI's o3. ARC-AGI-2 targets test-time reasoning.…

Thanks for your awesome work Greg!

The success of o3 directly contradicts us being in an "idea-constrained environment", what makes you believe that?

Re: Arc-AGI-2 and ARC Prize 2025

#16

Post by zamadatix »

gkamradt wrote: post #13
zamadatix wrote: post #11

Earlier quoted context omitted.

What prevents everything in 4 from becoming a part of 3 the first time the test set is run on a proprietary model, do you require competitors like OpenAI provide models Kaggle can self host for the test?

#4 (private test set) doesn't get used for any public model testing. It is only used on the Kaggle leaderboard where no internet access is allowed.

Sorry, I probably phrased the question poorly. My question is more along the lines of "when you already scored e.g. OpenAI's o3 on ARC AGI 2 how did you guarantee OpenAI can't just look at its server logs to see question set 4"?

Re: Arc-AGI-2 and ARC Prize 2025

#17

Post by ipunchghosts »

The computer vision community needs an dataset like this for evaluation... train in one domain and test on another. The best we have now are thr imagenet r and c datasets. Humans have no issues with domain adaptation with vision, but comouter vision models struggle in many ways sti including out of domain images.

Re: Arc-AGI-2 and ARC Prize 2025

#18

Post by gkamradt »

zamadatix wrote: post #16
gkamradt wrote: post #13

Earlier quoted context omitted.

#4 (private test set) doesn't get used for any public model testing. It is only used on the Kaggle leaderboard where no internet access is allowed.

Sorry, I probably phrased the question poorly. My question is more along the lines of "when you already scored e.g. OpenAI's o3 on ARC AGI 2 how did you guarantee OpenAI can't just look at its server logs to see question set 4"?

Ah yes, two things

1. We had a no-data retention agreement with them. We were assured by the highest level of their company + security division that the box our test was run on would be wiped after testing

2. We only tested o3 against the semi-private set. We didn't test it with the private eval.

Re: Arc-AGI-2 and ARC Prize 2025

#19

Post by zamadatix »

gkamradt wrote: post #18
zamadatix wrote: post #16

Earlier quoted context omitted.

Sorry, I probably phrased the question poorly. My question is more along the lines of "when you already scored e.g. OpenAI's o3 on ARC AGI 2 how did you guarantee OpenAI can't just look at its server logs to see question set 4"?

Ah yes, two things 1. We had a no-data retention agreement with them. We were assured by the highest level of their company + security division that the box our test was run on would be wiped after testing 2. We only tested o3 against the semi-private set. We didn't test it with the private eval.

Makes sense, particularly part 2 until "the final results" are needed. Thanks for taking the time to answer my question!

Re: Arc-AGI-2 and ARC Prize 2025

#20

Post by jmtulloss »

gmkhf wrote: post #7
gkamradt wrote: post #2

Hey HN, Greg from ARC Prize Foundation here. Alongside Mike Knoop and François Francois Chollet, we’re launching ARC-AGI-2, a frontier AI benchmark that measures a model’s ability to generalize on tasks it hasn’t seen before, and the ARC Prize 2025 competition to beat it. In Dec ‘24, ARC-AGI-1 (2019) pinpointed the moment AI moved beyond pure memorization as seen by OpenAI's o3. ARC-AGI-2 targets test-time reasoning.…

I think a lot of people got discouraged, seeing how openai solved arc agi 1 by what seems like brute forcing and throwing money at it. Do you believe arc was solved in the "spirit" of the challenge? Also all the open sourced solutions seem super specific to solving arc. Is this really leading us to human level AI at open ended tasks?

Why is this the same comment as https://news.ycombinator.com/item?id=43466406?
Post reply on HN