Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Day 1 of ARC-AGI-3

symbolica.ai

Post reply on HN
Print view View original

1–10 of 79 posts

Re: Day 1 of ARC-AGI-3

#4

Post by falcor84 »

lairv wrote: post #2

Note that this uses a harness so it doesn't qualify for the official ARC-AGI-3 leaderboard According to the authors the harness isn't ARC-AGI specific though https://x.com/agenticasdk/status/2037335806264971461

I for one think that harness development is perhaps the most interesting part at the moment and would love to have an alternative leaderboard with harnesses.

Re: Day 1 of ARC-AGI-3

#5

Post by sanxiyn »

falcor84 wrote: post #4
lairv wrote: post #2

Note that this uses a harness so it doesn't qualify for the official ARC-AGI-3 leaderboard According to the authors the harness isn't ARC-AGI specific though https://x.com/agenticasdk/status/2037335806264971461

I for one think that harness development is perhaps the most interesting part at the moment and would love to have an alternative leaderboard with harnesses.

There is. Official leaderboard is without harness, and community leaderboard is with harness. Read ARC-AGI-3 Technical Paper for details.

Re: Day 1 of ARC-AGI-3

#6

Post by falcor84 »

sanxiyn wrote: post #5
falcor84 wrote: post #4

Earlier quoted context omitted.

I for one think that harness development is perhaps the most interesting part at the moment and would love to have an alternative leaderboard with harnesses.

There is. Official leaderboard is without harness, and community leaderboard is with harness. Read ARC-AGI-3 Technical Paper for details.

I went through the technical paper again, and while they explain why they decided against the harness, I disagree with them - my take is that if harnesses are overfitting, then they should be penalized on the hidden test set.

Anyway, searching both in ARC-AGI's paper and website and directly on kaggle, I failed to find a with-harness leaderboard; can you please give the link?

Re: Day 1 of ARC-AGI-3

#7

Post by sanxiyn »

falcor84 wrote: post #6
sanxiyn wrote: post #5

Earlier quoted context omitted.

There is. Official leaderboard is without harness, and community leaderboard is with harness. Read ARC-AGI-3 Technical Paper for details.

I went through the technical paper again, and while they explain why they decided against the harness, I disagree with them - my take is that if harnesses are overfitting, then they should be penalized on the hidden test set. Anyway, searching both in ARC-AGI's paper and website and directly on kaggle, I failed to find a with-harness leaderboard; can you please give the link?

Here it is: https://arcprize.org/leaderboard/community

Re: Day 1 of ARC-AGI-3

#8

Post by krackers »

lairv wrote: post #2

Note that this uses a harness so it doesn't qualify for the official ARC-AGI-3 leaderboard According to the authors the harness isn't ARC-AGI specific though https://x.com/agenticasdk/status/2037335806264971461

> this uses a harness

This seems like an arbitrary restriction. Tool-use requires a harness, and their whitepaper never defines exactly what counts as valid.

Re: Day 1 of ARC-AGI-3

#10

Post by osti »

lairv wrote: post #2

Note that this uses a harness so it doesn't qualify for the official ARC-AGI-3 leaderboard According to the authors the harness isn't ARC-AGI specific though https://x.com/agenticasdk/status/2037335806264971461

Doesn't the chat version of chatgpt or gemini also have interleaved tool calls, so do those also count as with harnesses?
Post reply on HN