Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

François Chollet: The Arc Prize and How We Get to AGI [video]

youtube.com

Post reply on HN
Print view View original

161–170 of 230 posts

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#161

Post by goatlover »

tedy1996 wrote: post #132
lostphilosopher wrote: post #89

Earlier quoted context omitted.

We don't really have a true test that means "if we pass this test we have AGI" but we have a variety of tests (like ARC) that we believe any true AGI would be able to pass. It's a "necessary but not sufficient" situation. Also ties directly to the challenge in defining what AGI really means. You see a lot of discussions of "moving the goal posts" around AGI, but as I see it we've never had goal posts, we've just got…

I have graduated with a degree in Software engineering and i am bilingual (Bulgarian and English). Currently AI is better than me in everything except adding big numbers or writing code in really niche topics - for example code golfing a Brainfuck interpreter or writing a Rubiks cube solver. I believe AGI has been here for at least a year now.

Can the AI wash your dishes, fold your laundry, take out your trash, meet a friend for dinner or the other thousand things you might do in an average day when you're not interacting with text on a screen?

You know stuff that humans have done way before there were computers and screens.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#162

Post by gonzobonzo »

godshatter wrote: post #77
gonzobonzo wrote: post #67

Earlier quoted context omitted.

I agree with you but I'll go a step further - these benchmarks are a good example of how far we are from AGI. A good base test would be to give a manager a mixed team of remote workers, half being human and half being AI, and seeing if the manager or any of the coworkers would be able to tell the difference. We wouldn't be able to say that AI that passed that test would necessarily be AGI, since we would have to test…

The problem with "spot the difference" tests, imho, is that I would expect an AGI to be easily spotted. There's going to be a speed of calculation difference, at the very least. If nothing else, typing speed would be completely different unless the AGI is supposed to be deceptive. Who knows what it's personality would be like. I'd say it's a simple enough test just to see if an AGI could be hired as, for example, an…

Those could probably be mitigated pretty easily in testing situations. For example, making sure all participants had a delay in chat conversations, or running correspondence through an LLM to equalize the personality.

However, I'm not sure an AGI test should be mitigating them. If an AI isn't able to communicate at human speeds, or isn't able to achieve the social understandings that a human does, it would probably be wrong to say that it has the same intelligence capabilities as a human (how AGI has traditionally been defined). It wouldn't be able to provide human level performance in many jobs.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#163

Post by goatlover »

soVeryTired wrote: post #129
gtech1 wrote: post #124

This may be a silly question, I'm no expert. But why not simply define as AGI any system that can answer a question that no human can. So for example, ask AGI to find out, from current knowledge, how to reconcile gravity and qed.

Computers can already do a lot of things that no human can though. They can reliably find the best chess or go move better than a human. It's conceivable (though not likely) that given training enough training in symbolic mathematics and some experimental data, an LLM-style AI could figure out a neat reconciliation of the two theories. I wouldn't say that makes it AGI though. You could achieve that unification with a…

Wouldn't this unification need to be backed by empirical data? Let's say the AI discovers the two theories can be unified using let's say some configuration 8 spatial dimensions and 2 time dimensions. Neat trick, but how do we know the world actually has those dimensions?

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#164

Post by brcmthrowaway »

Bjorkbat wrote: post #119
hackinthebochs wrote: post #15

Earlier quoted context omitted.

Absolutely. Presumably there is some specific considerations or evidence that helped him evolve his opinion. I would be interested in seeing a writeup about it. With him having been a very public advocate against AGI, a writeup of his evolution seems appropriate and would be very edifying for a lot of people.

I recall it as less an evolution and more a complete tonal shift the moment o3 was evaluated on ARC-AGI. I remember on Twitter Sam made some dumb post suggesting they had beaten the benchmark internally and Francois calling him out on his vagueposting. Soon as they publicly released the scores, it was like he was all-in on reasoning. Which I have to admit I was kind of disappointed by.

What exactly is "reasoning"?

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#165

Post by umeshunni »

goatlover wrote: post #161
tedy1996 wrote: post #132

Earlier quoted context omitted.

I have graduated with a degree in Software engineering and i am bilingual (Bulgarian and English). Currently AI is better than me in everything except adding big numbers or writing code in really niche topics - for example code golfing a Brainfuck interpreter or writing a Rubiks cube solver. I believe AGI has been here for at least a year now.

Can the AI wash your dishes, fold your laundry, take out your trash, meet a friend for dinner or the other thousand things you might do in an average day when you're not interacting with text on a screen? You know stuff that humans have done way before there were computers and screens.

Yeah, I'm convinced that the biggest difference between the current generation of AIs we have and humans is that AIs don't have the range of tool use and interaction with the physical environment that humans do. And that's what's actually holding AGI back not access to more data.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#166

Post by whattheheckheck »

cubefox wrote: post #142
TheAceOfHearts wrote: post #20

Earlier quoted context omitted.

Getting a high score on ARC doesn't mean we have AGI and Chollet has always said as much AFAIK, it's meant to push the AI research space in a positive direction. Being able to solve ARC problems is probably a pre-requisite to AGI. It's a directional push into the fog of war, with the claim being that we should explore that area because we expect it's relevant to building AGI.

> Getting a high score on ARC doesn't mean we have AGI and Chollet has always said as much AFAIK He only seems to say this recently, since OpenAI cracked the ARC-AGI benchmark. But in the original 2019 abstract he said this: > We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans. https://arxiv.o…

Okay, true AGI would solve the coordination problem for all humans everywhere and usher in a post scarcity utopia. https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

We can simply check the news every day until it's built...

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#167

Post by bogtog »

krackers wrote: post #151
bogtog wrote: post #91

I wonder how much slow progress on ARC can be explained by their visual properties making them easy for humans but hard for LLMs. My impression is that models are pretty bad at interpreting grids of characters. Yesterday, I was trying to get Claude to convert a message into a cipher where it converted a 98-character string into 7x14 grid where the sequential letters moved 2-right and 1-down (i.e., like a knight it ch…

I thought so too back when the test was first released, but now that we have multimodal models which can take images directly as input, shouldn't this point be moot?

I think the top performer afaik (ChatGPT o3) is still treating ARC as a series of characters. I imagine complex reasoning in multimodal processing wouldn't be nearly as advanced so treating it as characters is still better

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#168

Post by gtech1 »

goatlover wrote: post #163
soVeryTired wrote: post #129

Earlier quoted context omitted.

Computers can already do a lot of things that no human can though. They can reliably find the best chess or go move better than a human. It's conceivable (though not likely) that given training enough training in symbolic mathematics and some experimental data, an LLM-style AI could figure out a neat reconciliation of the two theories. I wouldn't say that makes it AGI though. You could achieve that unification with a…

Wouldn't this unification need to be backed by empirical data? Let's say the AI discovers the two theories can be unified using let's say some configuration 8 spatial dimensions and 2 time dimensions. Neat trick, but how do we know the world actually has those dimensions?

Do we even have any other theory that does that already ? It seems that even finding one would be a great achievement

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#169

Post by krackers »

bogtog wrote: post #167
krackers wrote: post #151

Earlier quoted context omitted.

I thought so too back when the test was first released, but now that we have multimodal models which can take images directly as input, shouldn't this point be moot?

I think the top performer afaik (ChatGPT o3) is still treating ARC as a series of characters. I imagine complex reasoning in multimodal processing wouldn't be nearly as advanced so treating it as characters is still better

interesting, I thought one of the whole points of o3 was mixed multimodal reasoning (e.g. everyone doing those geoguesser challenges). But maybe that's just a parlor trick and it's not actually implemented that way. I wonder when they're going to extend chain-of-thought to work with image tokens, seems like that'd help for solving spatial challenges like this.

Re: François Chollet: The Arc Prize and How We Get to AGI [video]

#170

Post by tbrownaw »

singron wrote: post #141
ummonk wrote: post #98

Earlier quoted context omitted.

"Being able to solve ARC problems is probably a pre-requisite to AGI." - is it? Humans have general intelligence and most can't solve the harder ARC problems.

https://arcprize.org/leaderboard "Avg. Mturker" has 77% on ARC1 and costs $3/task. "Stem Grad" has 98% on ARC1 and costs $10/task. I would love a segment like "typical US office employee" or something else in between since I don't think you need a stem degree to do better than 77%. It's also worth noting the "Human Panel" gets 100% on ARC2 at $17/task. All the "Human" models are on the score/cost frontier and excepti…

> I think the real argument is that the ARC problems are too abstract and obscure to be relevant to useful AGI

I think it's meant to work like how getting things off the top shelf at the supermarket isn't relevant to playing basketball.

Post reply on HN