Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

François Chollet: The Arc Prize and How We Get to AGI [video]

youtube.com

Post reply on HN
Print view View original

31–40 of 230 posts

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#31

Post by TheAceOfHearts »

The first highlight from this video is getting to see a preview of the next ARC dataset. Otherwise it feels like most of what Chollet says here has already been repeated in his other podcast appearances and videos. It's a good video if you're not familiarized with his work, but if you've seen some of his recent interviews then you can probably skip the first 20 minutes.

The second highlight from this video is the section from 29 minutes onward, where he talks about designing systems that can build up rich libraries of abstractions which can be applied to new problems. I wish he had lingered more on exploring and explaining this approach, but maybe they're trying to keep a bit of secret sauce because it's what his company is actively working on.

One of the major points which seems to be emerging from recent AI discourse is that the ability to integrate continuous learning seems like it'll be a key element in building AGI. Context is fine for short tasks, but if lessons are never preserved you're severely capped with how far the system can go.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#32

Post by KoolKat23 »

oldge wrote: post #19
qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

Today’s llms are fancy autocomplete but lack test time self learning or persistent drive. By contrast, an AGI would require: – A goal-generation mechanism (G) that can propose objectives without external prompts – A utility function (U) and policy π(a│s) enabling action selection and hierarchy formation over extended horizons – Stateful memory (M) + feedback integration to evaluate outcomes, revise plans, and execute…

I'd say we're not far off.

Looking at the human side, it takes a while to actually learn something. If you've recently read something it remains in your "context window". You need to dream about it, to think about, to revisit and repeat until you actually learn it and "update your internal model". We need a mechanism for continuous weight updating.

Goal-generation is pretty much covered by your body constantly drip-feeding your brain various hormones "ongoing input prompts".

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#34

Post by maaaaattttt »

qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

I've said this somewhere else, but we have the perfect test for AGI in the form of any open world game. Give the instructions to the AGI that it should finish the game and how to control it. Give the frames as input and wait. When I think of the latest Zelda games and especially how the Shrine chanllenges are desgined they especially feel like the perfect environement for an AGI test.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#35

Post by saberience »

The Arc prize/benchmark is a terrible judge of whether we got to AGI.

If we assume that humans have "general intelligence", we would assume all humans could ace Arc... but they can't. Try asking your average person, i.e. supermarket workers, gas station attendants etc to do the Arc puzzles, they will do poorly, especially on the newer ones, but AI has to do perfectly to prove they have general intelligence? (not trying to throw shade here but the reality is this test is more like an IQ test than an AGI test).

Arc is a great example of AI researchers moving the goal posts for what we consider intelligent.

Let's get real, Claude Opus is smarter than 99% of people right now, and I would trust its decision making over 99% of people I know in most situations, except perhaps emotion driven ones.

Arc agi benchmark is just a gimmick. Also, since it's a visual test and the current models are text based it's actually a rigged (against the AI models) test anyway, since their datasets were completely text based.

Basically, it's a test of some kind, but it doesn't mean quite as much as Chollet thinks it means.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#36

Post by davidclark »

qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

In the video, François Chollet, creator of the ARC benchmarks, says that beating ARC does not equate to AGI. He specifically says they will be able to be beaten without AGI.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#37

Post by hackinthebochs »

cubefox wrote: post #29
hackinthebochs wrote: post #10

Has Chollet ever talked about his change of heart regarding AGI? It wasn't that long ago when he was one of the loudest voices decrying even the concept of AGI, let alone us being on the path to creating it. Now he's an advocate and has his own prize dataset? Seems rather convenient to change your tune once hundreds of billions are being thrown at AGI (not that I would blame him).

ARC-AGI was introduced in 2019: https://arxiv.org/abs/1911.01547 GPT-3 didn't come out until 2020.

In my view that just makes his evolution more interesting as it wasn't just a matter of being wow'ed by what ChatGPT could do.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#38

Post by 0xCE0 »

hackinthebochs wrote: post #10

Has Chollet ever talked about his change of heart regarding AGI? It wasn't that long ago when he was one of the loudest voices decrying even the concept of AGI, let alone us being on the path to creating it. Now he's an advocate and has his own prize dataset? Seems rather convenient to change your tune once hundreds of billions are being thrown at AGI (not that I would blame him).

He has recently co-founded NDEA company, so he has to align himself for that. Same kind of vibe change feels for Joscha Bach after having some position in Liquid AI company. Communication is not so relaxed anymore.

That said, I'd still listen these two guys (+ Schmidhuber) more than any other AI-guy.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#39

Post by autobodie »

TheAceOfHearts wrote: post #20
qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

Getting a high score on ARC doesn't mean we have AGI and Chollet has always said as much AFAIK, it's meant to push the AI research space in a positive direction. Being able to solve ARC problems is probably a pre-requisite to AGI. It's a directional push into the fog of war, with the claim being that we should explore that area because we expect it's relevant to building AGI.

[flagged]

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#40

Post by [deleted] »

autobodie wrote: post #39
TheAceOfHearts wrote: post #20

Earlier quoted context omitted.

Getting a high score on ARC doesn't mean we have AGI and Chollet has always said as much AFAIK, it's meant to push the AI research space in a positive direction. Being able to solve ARC problems is probably a pre-requisite to AGI. It's a directional push into the fog of war, with the claim being that we should explore that area because we expect it's relevant to building AGI.

[flagged]

[deleted]
Post reply on HN