Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

François Chollet: The Arc Prize and How We Get to AGI [video]

youtube.com

Post reply on HN
Print view View original

191–200 of 230 posts

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#191

Post by ACCount36 »

krackers wrote: post #151
bogtog wrote: post #91

I wonder how much slow progress on ARC can be explained by their visual properties making them easy for humans but hard for LLMs. My impression is that models are pretty bad at interpreting grids of characters. Yesterday, I was trying to get Claude to convert a message into a cipher where it converted a 98-character string into 7x14 grid where the sequential letters moved 2-right and 1-down (i.e., like a knight it ch…

I thought so too back when the test was first released, but now that we have multimodal models which can take images directly as input, shouldn't this point be moot?

Even the very best multimodal LLMs still suffer from a harsh perception bottleneck. They're impressive, but nowhere near as good as human visual cortex.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#192

Post by cubefox »

fragmede wrote: post #178
whattheheckheck wrote: post #166

Earlier quoted context omitted.

Okay, true AGI would solve the coordination problem for all humans everywhere and usher in a post scarcity utopia. https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ We can simply check the news every day until it's built...

That's ASI, artificial super intelligence, not AGI.

More precisely, it's value-aligned ASI.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#193

Post by staunton »

fvdessen wrote: post #137
MPSimmons wrote: post #130

Earlier quoted context omitted.

I don't think we actually even have a good definition of "This is what AGI is, and here are the stationary goal posts that, when these thresholds are met, then we will have AGI". If you judged human intelligence by our AI standards, then would humans even pass as Natural General Intelligence? Human intelligence tests are constantly changing, being invalidated, and rerolled as well. I maintain that today's modern LLMs…

Turing test is not really that meaningful anymore because you can always detect the AI by text and timing patterns rather than actual intelligence. In fact the most reliable way to test for AI is probably to ask trivia questions on various niche topics, I don't think any human has as much breath of general knowledge as current AIs.

> you can always detect the AI by text and timing patterns

I see no reason why an AI couldn't be trained on human data to fake all of that.

If noone has bothered so far, that's because pretty much all commercial applications of this would be illegal or at least leading to major reputational damage when exposed.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#194

Post by cubefox »

yorwba wrote: post #172
cubefox wrote: post #145

Earlier quoted context omitted.

> I think the people behind the ARC Prize agree that getting a high score doesn't mean we have AGI The benchmark was literally called ARC-AGI. Only after OpenAI cracked it, they started backtracking and saying that it doesn't test for true AGI. Which undermines the whole premise of a benchmark.

It does test for AGI. For its absence.

I would call something as intelligent as an animal an AGI. They are very general (as opposed to narrow) intelligences, just not necessarily very smart ones. ARC-AGI arguably conflates these two dimensions.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#195

Post by yunwal »

qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

The author of the test agrees with you.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#197

Post by kcaj »

From the talk: “Intelligence is the conversion ratio between past experience and potential operating area [where the potential operating area of interest is the area of problems that system has no experience of].”

So we might say, “General Intelligence is the ability to do the things we haven’t yet thought of.”

“Like what?”

“Well, as soon as I name something it stops counting.”

Gödellian - I like it. Does that mean a constructive definition of General Intelligence is uncomputable?

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#198

Post by roenxi »

tmvphil wrote: post #45
roenxi wrote: post #21

By both definitions of intelligence in the presentation we should be saying "how we got to AGI" in the past tense. We're already there. AI's can deal with situations they weren't prepared for in any sense that a human can. They might not do well, but they'll have a crack at it. We can trivially build systems that collect data and do a bit more offline training if that is what someone wants to see, but there doesn't r…

According to this presentation at least, ARC-AGI-2 shows that there is a big meaningful gap in fluid intelligence between normal non-genius humans and the best models currently, which seems to indicate we are not "already there".

If there is a gap in intelligence between two humans, does that mean to you that one of them is necessarily not a general intelligence? The current crop of AIs get some of the questions right by reasoning through them. That means they are already intelligent in the way ARC-AGI-2 measures intelligence. They just aren't very capable ones.

If AI at least equal humans in all intellectual fields then they are super-intelligences, because there are already fields where they dominate humans so outrageously there isn't a competition (nearly all fields, these days). Before they are superintelligences there is a phase where they are just AGIs, we've been in that phase for a while now. Artificial superintelligence is very exciting, but Artificial non-super Intelligence or AGI is here with us in the present.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#199

Post by tmvphil »

roenxi wrote: post #198
tmvphil wrote: post #45

Earlier quoted context omitted.

According to this presentation at least, ARC-AGI-2 shows that there is a big meaningful gap in fluid intelligence between normal non-genius humans and the best models currently, which seems to indicate we are not "already there".

If there is a gap in intelligence between two humans, does that mean to you that one of them is necessarily not a general intelligence? The current crop of AIs get some of the questions right by reasoning through them. That means they are already intelligent in the way ARC-AGI-2 measures intelligence. They just aren't very capable ones. If AI at least equal humans in all intellectual fields then they are super-intell…

You can define AGI however you want I suppose, but I would consider it achieved when AI can achieve at least about median human performance on all cognitive tasks. Obviously computers are useful well before this point, but it is clearly meaningful line in the sand, useful enough to merit having a dedicated name like "AGI". Constructed tasks like ARC-AGI simply quantify what everyone can already see, which is that current models cannot be used as a drop-in replacement for humans in most cases.

To me, superintelligence means specifically either dominating us in our highest intellectual accomplishments, i.e. math, science, philosophy or literally dominating us via subordinating or eliminating humans. Neither of these things have happened at all.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#200

Post by dfilppi »

qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

[dead]
Post reply on HN