191–200 of 394 posts
- Take a person who grew up playing video games. They'll pass these tests 100% without even breaking a sweat.
- BUT, put a grandmother who has never used a computer in front of this game, and she'll most likely fail completely. Just like an LLM.
As soon as models are "natively" trained on a massive dataset of these types of games, they'll easily adapt and start crushing these challenges.
This is not AGI at all.
This is a very good estimation of AGI. We give humans and AI the same input and measure the results. Kudos to ARC for creating these games. I really wonder why so many people fight against this. We know that AI is useful, we know that AI is researchful, but we want to know if they are what we vaguely define as intelligence. I’ve read the airplanes don’t use wings, or submarines don’t swim. Yes, but this is is not the…
I would imagine if you simply encoded the game in textual format and asked an LLM to come up with a series of moves, it would beat humans.
The problem here is more around perception than anything.
This is a very good estimation of AGI. We give humans and AI the same input and measure the results. Kudos to ARC for creating these games. I really wonder why so many people fight against this. We know that AI is useful, we know that AI is researchful, but we want to know if they are what we vaguely define as intelligence. I’ve read the airplanes don’t use wings, or submarines don’t swim. Yes, but this is is not the…
AGI’s 'general' is the wrong word, I thinkg. Humans aren’t general, we’re jagged. Strong in some areas, weak in others, and already surpassed in many domains. LLM are way past us at languages for instance. Calculators passed us at calculating, etc.
Let's be honest: we are giving LLMs and humans the exact same tasks, but are we putting them on an equal playing field? Specifically, do they have access to the same resources and behavioral strategies?
- LLMs don't have spatial reasoning.
- LLMs don't have a lifetime of video game experience starting from childhood.
- LLMs don't have working memory or the ability to actually "memorize" key parameters on the fly.
- LLMs don't have an internal "world model" (one that actively adapts to real-world context and the actual process of playing a game).
... I could go on, but I've outlined the core requirements for beating these tests above.
So, are we putting LLMs and humans in the same position? My answer is "no." We give them the same tasks, but their approach to solving them—let alone their available resources—is fundamentally different. Even Einstein wouldn't necessarily pass these tests on the first try. He’d first have to figure out how to use a keyboard, and then frantically start "building up new experience."
P.S. To quickly address the idea that LLMs and calculators are just "useful tools" that will never become AGI—I have some bad news there too. We differ from calculators architecturally; we run on entirely different "processors." But with LLMs, we are architecturally built the same way: it is a Neural Network that processes and makes decisions. This means our only real advantage over them is our baseline configuration and the list of "tools" connected to our neural network (senses, motor functions, etc.). To me, this means LLMs don't have any fundamental "architectural" roadblocks. We just have a head start, but their speed of evolution is significantly faster.
I'll probably be the skeptic here, but: - Take a person who grew up playing video games. They'll pass these tests 100% without even breaking a sweat. - BUT, put a grandmother who has never used a computer in front of this game, and she'll most likely fail completely. Just like an LLM. As soon as models are "natively" trained on a massive dataset of these types of games, they'll easily adapt and start crushing these c…
I've been a gamer for just about 40 years. Gaming is my "thing"
I found the challenges fun, but easy. Coming back and reading comments from people struggling with the games, my first thought was - yup definitely not a gamer.
My approach was to poke at the controls to suss the rules, then the actual solutions were really straightforward.
fwiw, I'm pretty dumb generally, but these kinds of puzzles are my jam.
Earlier quoted context omitted.
Something that surprises me about modern LLMs is that they're relatively smart yet lack consciousness. I used to believe that consciousness (e.g. a desire for self-preservation, intrinsic motivation) might be a necessary requirement for AGI/ASI, but it's increasingly looking like that may not be the case. If true, that's actually good news, since it makes the worst doomsday scenarios less likely.
How can you tell?
Earlier quoted context omitted.
We don't call a calculator intelligent. A calculator is extremely useful, but it is not intelligent. A computer is extremely useful, but it is not intelligent. Airplanes don't have wings, but they're damn sure useful, and also not intelligent. If LLMs cannot learn to beat not-that-difficult of games better than young teens, they are not intelligent. They are extremely useful. But they are not AGI. Words matter.
> If LLMs cannot learn to beat not-that-difficult of games better than young teens, they are not intelligent. I agree, with unresolved questions. Does it count if the LLM writes code which trains a neural network to play the game, and that neural network plays the game better than people do? Does that only count if the LLM tries that solution without a human prompting it to do so?
The idea that we haven't taught LLMs to come up with new answers... That doesn't even sound plausible. Just crank up the temperature, and an LLM will throw out so many ideas you'll exhaust yourself trying to sort through them.
So what haven't we taught LLMs?
- Have we not taught them to "filter"? We just haven't equipped them with experience and intuition, because we only feed them either "absolute fakes" or "verified facts." We don't feed them the actual path of problem-solving and research; those datasets simply don't exist.
- Have we not taught them to "double-check"? They are already excellent at verifying the credibility of our work.
- Have we not taught them to "defend" their ideas? They can justify ironclad logic and spot potentially "flaky" logic better than any human.
- Have we not taught them to "publish" and "present to the scientific community"? It's just that the previous steps aren't fully polished yet.
And if you look at the question of "creating completely new ideas" from this angle and in this level of detail... To me personally, it doesn't seem at all like LLMs are incapable of this kind of work.
We simply haven't taught them how to do it yet, purely because we don't have a sufficient volume of the right training materials.
I'll probably be the skeptic here, but: - Take a person who grew up playing video games. They'll pass these tests 100% without even breaking a sweat. - BUT, put a grandmother who has never used a computer in front of this game, and she'll most likely fail completely. Just like an LLM. As soon as models are "natively" trained on a massive dataset of these types of games, they'll easily adapt and start crushing these c…
had the same thought. I've been a gamer for just about 40 years. Gaming is my "thing" I found the challenges fun, but easy. Coming back and reading comments from people struggling with the games, my first thought was - yup definitely not a gamer. My approach was to poke at the controls to suss the rules, then the actual solutions were really straightforward. fwiw, I'm pretty dumb generally, but these kinds of puzzles…
Earlier quoted context omitted.
> If LLMs cannot learn to beat not-that-difficult of games better than young teens, they are not intelligent. I agree, with unresolved questions. Does it count if the LLM writes code which trains a neural network to play the game, and that neural network plays the game better than people do? Does that only count if the LLM tries that solution without a human prompting it to do so?
I disagree that LLMs cannot solve "unsolved problems." This is already happening, and at fundamental mathematical and medical levels (the fields that are the most demanding when it comes to quality). The idea that we haven't taught LLMs to come up with new answers... That doesn't even sound plausible. Just crank up the temperature, and an LLM will throw out so many ideas you'll exhaust yourself trying to sort through…
ARC is trying to test if LLMs can actually learn how to play the game.
Earlier quoted context omitted.
AGI’s 'general' is the wrong word, I thinkg. Humans aren’t general, we’re jagged. Strong in some areas, weak in others, and already surpassed in many domains. LLM are way past us at languages for instance. Calculators passed us at calculating, etc.
We don't call a calculator intelligent. A calculator is extremely useful, but it is not intelligent. A computer is extremely useful, but it is not intelligent. Airplanes don't have wings, but they're damn sure useful, and also not intelligent. If LLMs cannot learn to beat not-that-difficult of games better than young teens, they are not intelligent. They are extremely useful. But they are not AGI. Words matter.
???
I'll probably be the skeptic here, but: - Take a person who grew up playing video games. They'll pass these tests 100% without even breaking a sweat. - BUT, put a grandmother who has never used a computer in front of this game, and she'll most likely fail completely. Just like an LLM. As soon as models are "natively" trained on a massive dataset of these types of games, they'll easily adapt and start crushing these c…
My main criticism would be that it doesn’t seem like this test allows online learning, which is what humans do (over the scale of days to years). So in practice it may still collapse to what you point out, but not because the task is unsuited to showing AGI.