Measuring progress toward AGI: A cognitive framework
1–10 of 229 posts
Re: Measuring progress toward AGI: A cognitive framework
#2Cool that we are at a stage where it is meaningful to start measuring progress toward AGI. Something I am wondering on the philosophical side: are we ever going to be able to tell if the system really "understands" and "perceives" the world?
Re: Measuring progress toward AGI: A cognitive framework
#3> Perception: extracting and processing sensory information from the environment
> Generation: producing outputs such as text, speech and actions
> Attention: focusing cognitive resources on what matters
> Learning: acquiring new knowledge through experience and instruction
> Memory: storing and retrieving information over time
> Reasoning: drawing valid conclusions through logical inference
> Metacognition: knowledge and monitoring of one's own cognitive processes
> Executive functions: planning, inhibition and cognitive flexibility
> Problem solving: finding effective solutions to domain-specific problems
> Social cognition: processing and interpreting social information and responding appropriately in social situations
--------------------
I prefer:
a) working memory (hold & manipulate information in mind simultaneously)
b) processing speed (how quickly & efficiently execute basic cognitive operations, leaving more resources for complex tasks)
c) fluid intelligence (ability to reason through novel problems without relying on prior knowledge)
d) crystallized intelligence (accumulated knowledge and ability to apply learned skills)
e) attentional control / executive function (focus, suppress irrelevant information, switch between tasks, inhibit impulsive responses)
f) long-term memory and retrieval (ability to form strong associations and retrieve them fluently)
g) spatial / visuospatial reasoning (mental rotation, visualization, navigating abstract spatial relationships)
h) pattern recognition & inductive reasoning (this is the most primitive and universal expression of intelligence across species, the ability to extract regularities from noisy data, to generalized from examples to rules)
Re: Measuring progress toward AGI: A cognitive framework
#4Cool that we are at a stage where it is meaningful to start measuring progress toward AGI. Something I am wondering on the philosophical side: are we ever going to be able to tell if the system really "understands" and "perceives" the world?
We'll get as close as we can with anything else, like trying to decide if a given human really "understands" and "perceives" the world.
Re: Measuring progress toward AGI: A cognitive framework
#5The two guys from Google get to set the rules?
How will they measure wisdom or common sense (ability to make an exception)?
Re: Measuring progress toward AGI: A cognitive framework
#6Those are crowdsourced benchmarks. We're calling them "cognitive" and "AGI" now, though. It's similar to when they made a benchmark and called it "GDP".
To be clear, I think we've seen very fast progress, certainly faster than I would have expected, I'm not trying to peddle some "wall" rhetoric here, but I struggle to see how this isn't just the SWE-bench du jour.
Re: Measuring progress toward AGI: A cognitive framework
#7Cool that we are at a stage where it is meaningful to start measuring progress toward AGI. Something I am wondering on the philosophical side: are we ever going to be able to tell if the system really "understands" and "perceives" the world?
We'll get as close as we can with anything else, like trying to decide if a given human really "understands" and "perceives" the world.
I thought of this when I saw that the final criteria in the list is Social Understanding. Might be a lot of humans who can't measure up to sentience by these parameters! ;-)
(and I wonder what my ADHD friends would think of the Executive Function requirement as well...)
Re: Measuring progress toward AGI: A cognitive framework
#8It still seems like something is missing from all these frameworks.
I feel like an average human wouldn't pass some of these metrics yet they are "generally intelligent". On the other hand they also wouldn't pass a lot of the expert questions that AI is good at.
We're measuring something, and I think optimizing it is useful, I'd even say it is "intelligent" in some ways, but it doesn't seem "intelligent" in the same way that humans are.
Re: Measuring progress toward AGI: A cognitive framework
#9Those are crowdsourced benchmarks. We're calling them "cognitive" and "AGI" now, though. It's similar to when they made a benchmark and called it "GDP". To be clear, I think we've seen very fast progress, certainly faster than I would have expected, I'm not trying to peddle some "wall" rhetoric here, but I struggle to see how this isn't just the SWE-bench du jour.
AGI is defined now as "whatever makes 1 trillion dollars of profit".
Re: Measuring progress toward AGI: A cognitive framework
#10> Perception: extracting and processing sensory information from the environment > Generation: producing outputs such as text, speech and actions > Attention: focusing cognitive resources on what matters > Learning: acquiring new knowledge through experience and instruction > Memory: storing and retrieving information over time > Reasoning: drawing valid conclusions through logical inference > Metacognition: knowledg…
>a) working memory (hold & manipulate information in mind simultaneously)
What counts as 'in mind' is undefined. You can succeed by declaring anything manipulatable counts as in.
>c) fluid intelligence (ability to reason through novel problems without relying on prior knowledge)
reasoning presupposes the conclusion. Solve is better. When a solution is given you cannot declare it to be not a solution. People can and do argue about if a answer was arrived at by reasoning even when they agree on the correctness.
>g) spatial / visuospatial reasoning (mental rotation, visualization, navigating abstract spatial relationships)
I have aphantasia, why should you exclude something from being intelligent because it cannot do something that I also cannot do.