Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

François Chollet: The Arc Prize and How We Get to AGI [video]

youtube.com

Post reply on HN
Print view View original

91–100 of 230 posts

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#91

Post by bogtog »

I wonder how much slow progress on ARC can be explained by their visual properties making them easy for humans but hard for LLMs.

My impression is that models are pretty bad at interpreting grids of characters. Yesterday, I was trying to get Claude to convert a message into a cipher where it converted a 98-character string into 7x14 grid where the sequential letters moved 2-right and 1-down (i.e., like a knight it chess). Claude seriously struggled.

Yet, Francois always pumps up the "fluid intelligence" component of this test and emphasizes how easy these are for humans. Yet, humans would presumably be terrible at the tasks if they looked at it character-by-character

This feels like a somewhat similar (intuition-lie?) case as the Apple paper showing how reasoning model's can't do tower of hanoi past 10+ disks. Readers will intuitively think about how they themselves could tediously do an infinitely long tower of hanoi, which is what the paper is trying to allude to. However, the more appropriate analogy would be writing out all >1000 moves on a piece of paper at once and being 100% correct, which is obviously much harder

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#92

Post by kelseyfrog »

FeepingCreature wrote: post #76
onemoresoop wrote: post #44

Earlier quoted context omitted.

> I'd say we're not far off. How are we not far off? How can LLMs generate goals and based on what?

You just train it on the goal. Then it has that goal. Alternately, you can train it on following a goal and then you have a system where you can specify a goal. At sufficient scale, a model will already contain goal-following algorithms because those help predict the next token when the model is basetrained on goal-following entities, ie. humans. Goal-driven RL then brings those algorithms to prominence.

How do you figure goal generation and supervised goal training are interchangeable?

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#93

Post by [deleted] »

wat10000 wrote: post #84
maaaaattttt wrote: post #56

Earlier quoted context omitted.

Given your premise (which I agree with) I think the issue in general comes from the lack of a good, broadly accepted definition of what AGI is. My initial comment originates from the fact that in my internal definition, an AGI would have a de facto understanding of the physics of "our world". Or better, could infer them by trial and error. But, indeed, it doesn't have to be the case. (The other advantage of the Zelda…

I'd say the issue is the lack of a good, broadly accepted definition of what I is. We all know "smart" when we see it, but actually defining it in a rigorous way is tough.

[deleted]

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#95

Post by ben_w »

kypro wrote: post #55
ben_w wrote: post #24

Earlier quoted context omitted.

You're not alone in this; I expect us to have not yet enumerated all the things that we ourselves mean by "intelligence". But conversely, not passing this test is a proof of not being as general as a human's intelligence.

I find the "what is intelligence?" discussion a little pointless if I'm honest. It's similar to asking a question like does it mean to be a "good person" and would we know whether an AI or person is really "good"? While understanding why a person or AI is doing what it's doing can be important (perhaps specifically in safety contexts) at the end of the day all that's really going to matter to most people is the outco…

That's certainly a valid way of looking at their abilities at any given task — "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim".

But when the question is "are they going to more important to the economy than humans?", then they have to be good at basically everything a human can do, otherwise we just see a variant of Amdahl's law in action and the AI perform an arbitrary speed-up of n % of the economy while humans are needed for the remaining 100-n %.

I may be wrong, but it seems to me that the ARC prize is more about the latter.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#96

Post by [deleted] »

oldge wrote: post #19
qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

Today’s llms are fancy autocomplete but lack test time self learning or persistent drive. By contrast, an AGI would require: – A goal-generation mechanism (G) that can propose objectives without external prompts – A utility function (U) and policy π(a│s) enabling action selection and hierarchy formation over extended horizons – Stateful memory (M) + feedback integration to evaluate outcomes, revise plans, and execute…

[deleted]

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#98

Post by ummonk »

TheAceOfHearts wrote: post #20
qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

Getting a high score on ARC doesn't mean we have AGI and Chollet has always said as much AFAIK, it's meant to push the AI research space in a positive direction. Being able to solve ARC problems is probably a pre-requisite to AGI. It's a directional push into the fog of war, with the claim being that we should explore that area because we expect it's relevant to building AGI.

"Being able to solve ARC problems is probably a pre-requisite to AGI." - is it? Humans have general intelligence and most can't solve the harder ARC problems.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#99

Post by mindcrime »

qoez wrote: post #4

I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI. It's mostly about pattern matching (and some of it ambiguous even for humans what the actual true response aught to be). It's like how in humans there's lots of different 'types' of intelligence, and just overfitting on IQ tests doesn't in my mind convince me a person is actually that smart.

> I feel like I'm the only one who isn't convinced getting a high score on the ARC eval test means we have AGI.

Wait, what? Approximately nobody is claiming that "getting a high score on the ARC eval test means we have AGI". It's a useful eval for measuring progress along the way, but I don't think anybody considers it the final word.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#100

Post by mNovak »

modeless wrote: post #78

ARC-AGI 3 remindes me of PuzzleScript games: https://www.puzzlescript.net/Gallery/index.html There are dozens of ready-made, well-designed, and very creative games there. All are tile-based and solved with only arrow keys and a single action button. Maybe someone should make a PuzzleScript AGI benchmark?

This game is great!

https://nebu-soku.itch.io/golfshall-we-golf

Maybe someone can make an MCP connection for the AIs to practice. But I think the idea of the benchmark is to reserve some puzzles for private evaluation, so that they're not in the training data.

Post reply on HN