[dead]
ARC-AGI-3
201–210 of 394 posts
Re: ARC-AGI-3
#202Perhaps actual AGI will be when the models create ARC-HGI-1 to test if humans have general intelligence.
Re: ARC-AGI-3
#203I'll probably be the skeptic here, but: - Take a person who grew up playing video games. They'll pass these tests 100% without even breaking a sweat. - BUT, put a grandmother who has never used a computer in front of this game, and she'll most likely fail completely. Just like an LLM. As soon as models are "natively" trained on a massive dataset of these types of games, they'll easily adapt and start crushing these c…
Isn’t this what AGI is by design? People CAN learn to become good at videogames. Modern LLMs can’t, they have to be retrained from scratch (I consider pre-training to be a completely different process than learning). I also don’t necessarily agree that a grandma would fail. Give her enough motivation and a couple days and she’ll manage these. My main criticism would be that it doesn’t seem like this test allows onlin…
What I'm saying is that this test is just another "out-of-distribution task" for an LLM. And it will be solved using the exact same methods we always use: it will end up in the pre-training data, and LLMs will crush it.
This has absolutely nothing to do with AGI. Once they beat these tests, new ones will pop up. They'll beat those, and people will invent the next batch.
The way I see it, the true formula for AGI is: [Brain] + [External Sensors] (World Receptors) + [Internal State Sensors] + [Survival Function] + [Memory].
I won't dive too deep into how each of these components has its own distinct traits and is deeply intertwined with the others (especially the survival function and memory). But on a fundamental level, my point is that we are not going to squeeze AGI out of LLMs just by throwing more tests and training cycles at them.
These current benchmarks aren't bringing us any closer to AGI. They merely prove that we've found a new layer of tasks that we simply haven't figured out how to train LLMs on yet.
P.S. A 2-year-old child is already an AGI in terms of its functional makeup and internal interaction architecture, even though they are far less equipped for survival than a kitten. The path to AGI isn't just endless task training—it's a shift toward a fundamentally different decision-making architecture.
Re: ARC-AGI-3
#204Ew. Cool demo, what idiot thought it was ok to have a half second cooldown between inputs? If I hit up three times I should move up three steps, not two steps because I pressed too quickly.
Re: ARC-AGI-3
#205Earlier quoted context omitted.
Francois here. The scoring metric design choices are detailed in the technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf - the metric is meant to discount brute-force attempts and to reward solving harder levels instead of the tutorial levels. The formula is inspired by the SPL metric from robotics navigation, it's pretty standard, not a brand new thing. We tested ~500 humans over 90 minute se…
Thanks, I mostly agree with your approach except for one thing: eyesight feels like a "harness" that humans get to use and LLMs do not. I'm guessing you did not pass the human testers JSON blobs to work with, and suspect they would also score 0% without the eyesight and visual cortex harness to their reasoning ability.
My sense is that a powerful enough AI would have the sense to think something like "ah, this sounds like a video game! Let me code up an interactive GUI, test it for myself, then use it to solve these puzzles..." and essentially self-harness (the way you would if you were reading a geometry problem, by drawing it out on paper).
Re: ARC-AGI-3
#206Earlier quoted context omitted.
Francois here. The scoring metric design choices are detailed in the technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf - the metric is meant to discount brute-force attempts and to reward solving harder levels instead of the tutorial levels. The formula is inspired by the SPL metric from robotics navigation, it's pretty standard, not a brand new thing. We tested ~500 humans over 90 minute se…
Suppose you construct a Mechanical Turk AI who plays ARC-AGI-3 by, for each task, randomly selecting one of the human players who attempted it, and scoring them as an AI taking those same actions would be scored. What score does this Turk get? It must be <100% since sometimes the random human will take more steps than the second best, but without knowing whether it's 90% or 50% it's very hard for me to contextualize…
The people recruited weren’t experts. I can imagine it’s straightforward to find humans (such as those that play many video games) that can score >100% on this benchmark.
Re: ARC-AGI-3
#207Earlier quoted context omitted.
How can you tell?
How can I tell what? That current LLMs are not conscious or that AGI/ASI will not require consciousness?
How do you know they aren't conscious of we don't know what consciousness is, and have no test to see if anyone or anything is conscious?
This may seem like a joke, but your answer will likely be in the vain of "conscious things are obviously conscious", which gets us nowhere.
I mean, self motivation and a desire to not be turned off can be programmed into even decades old AIs.
Re: ARC-AGI-3
#208I’m not a Chollet booster. Well, I might be a little bit of one in that I admire his persistence.
I really like these puzzles. There’s a lot to them both in design and scoring — models trained to do well on these are going to be genuinely much more useful, so I’m excited about it. As opposed to -1 and -2, to do well at these, you need to be able to do:
- Visual reasoning
- Path planning (and some fairly long paths)
- Mouse/screen interaction
- color and shape analysis
- cross-context learning/remembering
Probably more, I only did like five or six of these. We really want models that are good at all this; it covers a lot of what current agentic loops are super weak at. So I hope M. Chollet is successful at getting frontier labs to put a billion or so into training for these.
Re: ARC-AGI-3
#209Earlier quoted context omitted.
AGI’s 'general' is the wrong word, I thinkg. Humans aren’t general, we’re jagged. Strong in some areas, weak in others, and already surpassed in many domains. LLM are way past us at languages for instance. Calculators passed us at calculating, etc.
We don't call a calculator intelligent. A calculator is extremely useful, but it is not intelligent. A computer is extremely useful, but it is not intelligent. Airplanes don't have wings, but they're damn sure useful, and also not intelligent. If LLMs cannot learn to beat not-that-difficult of games better than young teens, they are not intelligent. They are extremely useful. But they are not AGI. Words matter.
So your definition of intelligence would be exactly equal to a human or some subset of them you choose? Could a dog solve ARC-AGI? Probably not. I would not say they lack intelligence. Same with a fruit fly. What if the calculator is powered by actual living neurons? I think you need to know where you actually think the difference between organic machine and intelligence is before making blanket statements.
A modern LLM in a loop with a harness for memory and behavior modification in a body would probably fool me.
Re: ARC-AGI-3
#210Earlier quoted context omitted.
How can I tell what? That current LLMs are not conscious or that AGI/ASI will not require consciousness?
How do you know they aren't conscious of we don't know what consciousness is, and have no test to see if anyone or anything is conscious? This may seem like a joke, but your answer will likely be in the vain of "conscious things are obviously conscious", which gets us nowhere. I mean, self motivation and a desire to not be turned off can be programmed into even decades old AIs.
As a philosophical zombie myself[0], I'm well aware of how hard it is to define and test consciousness. That's why I tried to clarify what I meant with: desire for self-preservation and intrinsic motivation. Which LLMs clearly lack, don't you agree? Also, I'm not saying that those things couldn't be programmed in, just that so far, they don't seem necessary.
[0] I lack a conscious experience and qualia