My takeaway from playing a number of levels is that I am definitely not AGI
ARC-AGI-3
11–20 of 394 posts
Re: ARC-AGI-3
#12Any benchmarks?
Re: ARC-AGI-3
#13So ARC-AGI was released in 2019. That's been solved, then there was ARC-AGI-2, and now there's ARC-AGI-3. What is even the point? Will ARC-AGI-26 hit the front page of Hacker News in 2057 ?
This is clear AGI progress. It should show you, that AI is not sleeping, it gets better and you should use this as a signal that you should take this topic serious.
Labelling a test "AGI" does not show AGI progress any more than labelling a cpu "AGI" makes it so. It might show that AI tools are improving but it does not necessarily follow that tools improving = AGI progress if you're on the completely wrong trail.
Re: ARC-AGI-3
#14what is the evidence that being able to play games equates to AGI?
Re: ARC-AGI-3
#15what is the evidence that being able to play games equates to AGI?
That is not the claim. It is a necessary condition, but not a sufficient one.
Re: ARC-AGI-3
#16i feel bad that we make the LLMs play this
Re: ARC-AGI-3
#17what is the evidence that being able to play games equates to AGI?
[deleted]
Re: ARC-AGI-3
#18what is the evidence that being able to play games equates to AGI?
There isn't a strict definition of AGI, there's no way to find evidence for what equates to it, and besides, things like this are meant only as likely necessary conditions.
Anyway, from the article:
> As long as there is a gap between AI and human learning, we do not have AGI.
This seems like a reasonable requirement. Something I think about a lot with vibe coding is that unlike humans, individual models do not get better within a codebase over time, they get worse.
Re: ARC-AGI-3
#19So ARC-AGI was released in 2019. That's been solved, then there was ARC-AGI-2, and now there's ARC-AGI-3. What is even the point? Will ARC-AGI-26 hit the front page of Hacker News in 2057 ?
You’re absolutely right to point it out.
LLMs weren’t supposed to solve 1, they did, so we got 2 and it really wasn’t supposed to be solvable by LLMs. It was, and as soon as it started creeping up we start hearing about 3: It’s Really AGI This Time.
I don’t know what Francois’ underlying story is, other than he hasn’t told it yet.
One of a few moments that confirmed it for me was when he was Just Asking Questions re: if Anthropic still used SaaS a month ago, which was an odd conflation of a hyperbolic reading of a hyperbolic stonk market bro narrative (SaaS is dead) and low-info on LLMs (Claude’s not the only one that can code) and addressing the wrong audience (if you follow Francois, you’re likely neither of those poles)
At this point I’d be more interested in a write up from Francois about where he is intellectually than an LLM that got 100% on this. It’s like when Yann would repeat endlessly that LLMs are definitionally dumber than housecats. Maybe, in some specific way that makes sense to you. You’re brilliant. But there’s a translation gap between Mount Olympus and us plebes, and you’re brilliant enough to know that too. So it comes across as trolling and boring.
Re: ARC-AGI-3
#20what is the evidence that being able to play games equates to AGI?
None whatsoever.
It's a "let's find a task humans are decent at, but modern AIs are still very bad at" kind of adversarial benchmark.
The exact coverage of this one is: spatial reasoning across multiple turns, agentic explore/exploit with rule inference and preplanning. Directly targeted against the current generation of LLMs.