Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC-AGI-3

arcprize.org

Post reply on HN
Print view View original

301–310 of 394 posts

Re: ARC-AGI-3

#301

Post by vonneumannstan »

mvkel wrote: post #212

Was just at the YC launch event for this. Haven't felt this much inspiration in a while. Incredible minds confronting on tech that will change our society. I met a guy who, for fun, started working on ARC2, and as he got the number to go up in the eval, a novel way to more efficiently move a robotic arm emerged. All that to say: chasing evals per se can have tangible real world benefits. Talking to the ARC folks toni…

>Talking to the ARC folks tonight, it sounds like there will be an ARC-4,5,6,etc. I mean of course there will be.

Quintessential goal post moving...

Re: ARC-AGI-3

#302

Post by daveguy »

dyauspitr wrote: post #136
Tiberium wrote: post #35

https://x.com/scaling01 has called out a lot of issues with ARC-AGI-3, some of them (directly copied from tweets, with minimal editing): - Human baseline is "defined as the second-best first-run human by action count". Your "regular people" are people who signed up for puzzle solving and you don't compare the score against a human average but against the second best human solution - The scoring doesn't tell you how m…

If anything this makes the test much harder for the LLM to get high scores and that makes the scores they’re getting all that much more impressive.

The scroes they're getting are on the order of 0-1% for this ARC-AGI-3 benchmark.

Re: ARC-AGI-3

#303

Post by j1000 »

I feel like AGI test would be sense of humor. Somehow I cannot force any LLM to output any even normal level joke.

Re: ARC-AGI-3

#304

Post by naasking »

girvo wrote: post #124
fc417fc802 wrote: post #94

Earlier quoted context omitted.

Those are supposed to be issues? After reading your list my impression of ARC-AGI has gone up rather than down. All of those things seem like the right way to go about this.

Yeah I'm quite surprised as to how all of those are supposed to be considered problems. They all make sense to me if we're trying to judge whether these tools are AGI, no?

> They all make sense to me if we're trying to judge whether these tools are AGI, no?

As long as the mean and median human scores are clearly communicated, the scoring is fine. I think the human scores above would surprise people at first glance, even if they make sense once you think about it, so there's an argument to be made that scores can be misleading.

Re: ARC-AGI-3

#305

Post by vonneumannstan »

>As long as there is a gap between AI and human learning, we do not have AGI.

This is an absurd constraint. You could have a vastly superhuman AI that doesn't learn as efficiently as a human and it would not pass this definition while it simultaneously goes on to colonize the galaxy...

Re: ARC-AGI-3

#306

Post by tantalor »

tantalor wrote: post #287

The controls just feel really bad. The inputs are too small, and there is way too much lag.

I just checked, the size of the controls are 28x28

The minimum recommended size for mobile is 44x44

Re: ARC-AGI-3

#307

Post by Real_Egor »

suddenlybananas wrote: post #267
Real_Egor wrote: post #193

Earlier quoted context omitted.

I’d actually focus on something else entirely here. Let's be honest: we are giving LLMs and humans the exact same tasks, but are we putting them on an equal playing field? Specifically, do they have access to the same resources and behavioral strategies? - LLMs don't have spatial reasoning. - LLMs don't have a lifetime of video game experience starting from childhood. - LLMs don't have working memory or the ability t…

>But with LLMs, we are architecturally built the same way: it is a Neural Network that processes and makes decisions. There are high-level similarities between ANNs and the human brain but they are very, very, very different in a ton of ways.

[dead]

Re: ARC-AGI-3

#308

Post by daveguy »

Imnimo wrote: post #211
thereitgoes456 wrote: post #206

Earlier quoted context omitted.

The people recruited weren’t experts. I can imagine it’s straightforward to find humans (such as those that play many video games) that can score >100% on this benchmark.

So, if you look at the way the scoring works, 100% is the max. For each task, you get full credit if you solve in a number of steps less than or equal to the baseline. If you solve it with more steps, you get points off. But each task is scored independently, and you can't "make up" for solving one slowly by solving another quickly. Like suppose there were only two tasks, each with a baseline score of solving in 100…

The purpose is to benchmark both generality and intelligence. "Making up for" a poor score on one test with an excellent score on another would be the opposite of generality. There's a ceiling based on how consistent the performance is across all tasks.

Re: ARC-AGI-3

#309

Post by daveguy »

pawelk411 wrote: post #240
fc417fc802 wrote: post #171

Earlier quoted context omitted.

So what? Are you suggesting that an agent exhibiting genuine AGI will be tripped up by having to ingest json rather than rgb pixels? LLMs are largely trained on textual data so json is going to be much closer to whatever native is for them. But by all means, give the agents access to an API that returns pixel data. However I fully expect that would reduce performance rather than increase it.

Because it is. Opus 4.6 jumps from 0.0% to 97.1% when given visual input

Source? I haven't seen anything like that for ARC-AGI performance.

Also, if it makes that big of a difference, then make a renderer for your agent that looks like the web page and have it solve them in the graphical interface and funnel the results to the API. I guarantee you won't get better performance, because the AGI is going to have to "understand" the raw data can be represented as a 2D matrix regardless of whether it gets a 2D matrix of pixels or a 2D matrix of enumeration in JSON. If anything, that makes it a more difficult problem for a AI system that "speaks" in tokens.

Re: ARC-AGI-3

#310

Post by mvkel »

vonneumannstan wrote: post #301
mvkel wrote: post #212

Was just at the YC launch event for this. Haven't felt this much inspiration in a while. Incredible minds confronting on tech that will change our society. I met a guy who, for fun, started working on ARC2, and as he got the number to go up in the eval, a novel way to more efficiently move a robotic arm emerged. All that to say: chasing evals per se can have tangible real world benefits. Talking to the ARC folks toni…

>Talking to the ARC folks tonight, it sounds like there will be an ARC-4,5,6,etc. I mean of course there will be. Quintessential goal post moving...

If you read the charter of the eval (or any eval, really), this statement is pretty silly.

The whole point of each eval version is to identify a chunk of challenges that humans do well that AI can't. When AI gets to ~80, you move to the next chunk. When you run out of challenges, you have AGI.

Post reply on HN