Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

François Chollet: The Arc Prize and How We Get to AGI [video]

youtube.com

Post reply on HN
Print view View original

61–70 of 230 posts

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#61

Post by CamperBob2 »

qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

If you can write code to solve ARC by "overfitting," then give it a shot! There's prize money to be won, as long as your model does a good job on the hidden test set. Zuckerberg is said to be throwing around 8-figure signing bonuses for talent like that.

But then, I guess it wouldn't be "overfitting" after all, would it?

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#62

Post by AIPedant »

cubefox wrote: post #23
roenxi wrote: post #21

By both definitions of intelligence in the presentation we should be saying "how we got to AGI" in the past tense. We're already there. AI's can deal with situations they weren't prepared for in any sense that a human can. They might not do well, but they'll have a crack at it. We can trivially build systems that collect data and do a bit more offline training if that is what someone wants to see, but there doesn't r…

Well, there is also robotics, active inference, online learning, etc. Things animals can do well.

Current robots perform very badly on my patented and highly scientific ROACH-AGI benchmark - "is this thing smarter at navigating unfamiliar 3D spaces than a cockroach?"

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#63

Post by profchemai »

saberience wrote: post #35

The Arc prize/benchmark is a terrible judge of whether we got to AGI. If we assume that humans have "general intelligence", we would assume all humans could ace Arc... but they can't. Try asking your average person, i.e. supermarket workers, gas station attendants etc to do the Arc puzzles, they will do poorly, especially on the newer ones, but AI has to do perfectly to prove they have general intelligence? (not tryi…

Out of 100 of evals, ARC is a very distinct and unique eval, most frontier models are also visual now, don't see the harm in having this instead of another text eval.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#64

Post by saberience »

tmvphil wrote: post #45
roenxi wrote: post #21

By both definitions of intelligence in the presentation we should be saying "how we got to AGI" in the past tense. We're already there. AI's can deal with situations they weren't prepared for in any sense that a human can. They might not do well, but they'll have a crack at it. We can trivially build systems that collect data and do a bit more offline training if that is what someone wants to see, but there doesn't r…

According to this presentation at least, ARC-AGI-2 shows that there is a big meaningful gap in fluid intelligence between normal non-genius humans and the best models currently, which seems to indicate we are not "already there".

There's already a big meaningful gap between the things AIs can do which humans can't, so why do you only count as "meaningful" the things humans can do which AIs can't?

I enjoy seeing people repeatedly move the goalposts for "intelligence" as AIs simply get smarter and smarter every week. Soon AI will have to beat Einstein in Physics, Usain Bolt in running, and Steve Jobs in marketing to be considered AGI...

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#65

Post by tsurba »

NetRunnerSu wrote: post #49
onemoresoop wrote: post #44

Earlier quoted context omitted.

> I'd say we're not far off. How are we not far off? How can LLMs generate goals and based on what?

Minimize prediction errors.

But are we close to doing that in real-time on any reasonably large model? I don’t think so.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#66

Post by jcranmer »

Lerc wrote: post #41
maaaaattttt wrote: post #34

Earlier quoted context omitted.

I've said this somewhere else, but we have the perfect test for AGI in the form of any open world game. Give the instructions to the AGI that it should finish the game and how to control it. Give the frames as input and wait. When I think of the latest Zelda games and especially how the Shrine chanllenges are desgined they especially feel like the perfect environement for an AGI test.

And if someone makes a machine that does all that and another person says "That's not really AGI because xyz" What then? The difficulty in coming up with a test for AGI is coming up with something that people will accept a passing grade as AGI. In many respects I feel like all of the claims that models don't really understand or have internal representation or whatever tend to lean on nebulous or circular definitions…

> The difficulty in coming up with a test for AGI is coming up with something that people will accept a passing grade as AGI.

The difficulty with intelligence is we don't even know what it is in the first place (in a psychology sense, we don't even have a reliable model of anything that corresponds to what humans point at and call intelligence; IQ and g are really poor substitutes).

Add into that Goodhart's Law (essentially, propose a test as a metric for something, and people will optimize for the test rather than what the test is trying to measure), and it's really no surprise that there's no test for AGI.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#67

Post by gonzobonzo »

qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

I agree with you but I'll go a step further - these benchmarks are a good example of how far we are from AGI.

A good base test would be to give a manager a mixed team of remote workers, half being human and half being AI, and seeing if the manager or any of the coworkers would be able to tell the difference. We wouldn't be able to say that AI that passed that test would necessarily be AGI, since we would have to test it in other situations. But we could say that AI that couldn't pass that test wouldn't qualify, since it wouldn't be able to successfully accomplish some tasks that humans are able to.

But of course, current AI is nowhere near that level yet. We're left with benchmarks, because we all know how far away we are from actual AGI.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#68

Post by jacquesm »

Let's not. Seriously. I absolutely love François and have used his work extensively. But looking around me at the social impact of AI I am really not convinced that this is what the world needs right now and that if we can stave off the turning point for another decade or two that humanity will likely benefit from that. The last thing we need is to inject yet another instability into a planet that is already fighting existential crisis on a number of fronts.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#69

Post by SubiculumCode »

qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

[1]https://app.rescript.info/public/share/W_T7E1OC2Wj49ccqlIOOz...

Perhaps it's because the representations are fractured. The link above is to the transcript of an episode of Machine Learning Street Talk with Kenneth O. Stanleyabout The Fractured Entangled Representation Hypothesis[1]

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#70

Post by thatguy0900 »

jacquesm wrote: post #68

Let's not. Seriously. I absolutely love François and have used his work extensively. But looking around me at the social impact of AI I am really not convinced that this is what the world needs right now and that if we can stave off the turning point for another decade or two that humanity will likely benefit from that. The last thing we need is to inject yet another instability into a planet that is already fighting…

It doesn't matter what should or should not happen. Technology will continue to race forward at breakneck speed while everyone involved pats each other on the back for making a bunch of money before the consequences hit
Post reply on HN