Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC Prize – a $1M+ competition towards open AGI progress

arcprize.org

Post reply on HN
Print view View original

41–50 of 351 posts

Re: ARC Prize – a $1M+ competition towards open AGI progress

#41

Post by bbor »

VirusNewbie wrote: post #36
salamo wrote: post #21

This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and ha…

>: humans do not need 10,000 examples to tell the difference between cats and dogs well, maybe. We view things in three dimensions at high fidelity: viewing a single dog or cat actually ends up being thousands of training samples, no?

Eh, still doesn’t hold up. I really don’t think there’s many psychologists working on the posited mechanism of simple NN-like backprop learning. Aka conditioning, I guess. As Chomsky reminds us every time we let him: human children learn to understand and use language — an incredibly complex and nuanced domain, to say the least — with shockingly little data and often zero-to-none intentional instruction. We definitely employ principles and patterns that are far more complex (more “emergent”?) than linear regression.

Tho I only ever did undergrad stats, maybe ML isn’t even technically a linear regression at this point. Still, hopefully my gist is clear

Re: ARC Prize – a $1M+ competition towards open AGI progress

#42

Post by kenjackson »

pmayrgundter wrote: post #14

This claim that these tests are easy for humans seems dubious, and so I went looking a bit. Melanie Mitchell chimed in on Chollet's thread and posted their related test [ConceptARC]. In it they question the ease of Chollet's tests: "One limitation on ARC’s usefulness for AI research is that it might be too challenging. Many of the tasks in Chollet’s corpus are difficult even for humans, and the corpus as a whole migh…

I just tried the first puzzle and I can't get it right. I think my solution makes logical sense and I explain why the patterns are consistent with the input, but it says its wrong. I'm either a lot dumber than I thought or they need to do a better job of vetting their tests.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#43

Post by modeless »

lacker wrote: post #6

I really like the idea of ARC. But to me the problems seem like they require a lot of spatial world knowledge, more than they require abstract reasoning. Shapes overlapping each other, containing each other, slicing up and reassembling pieces, denoising regular geometric shapes, you can call them "core knowledge" but to me it seems like they are more like "things that are intuitive to human visual processing". Would…

> to me it seems like they are more like "things that are intuitive to human visual processing".

Yann LeCun argues that humans are not general intelligence and that such a thing doesn't really exist. Intelligence can only be measured in specific domains. To the extent that this test represents a domain where humans greatly outperform AI, it's a useful test. We need more tests like that, because AIs are acing all of our regular tests despite being obviously less capable than humans in many domains.

> the problems require spatial knowledge that we intelligent humans learn with far more than 800 training examples.

Pretraining on unlimited amounts of data is fair game. Generalizing from readily available data to the test tasks is exactly what humans are doing.

> Would an intelligent but blind human be able to solve these problems?

I'm confident that they would, given a translation of the colors to tactile sensation. Blind humans still understand spatial relationships.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#44

Post by HarHarVeryFunny »

nickpsecurity wrote: post #7
lacker wrote: post #6

I really like the idea of ARC. But to me the problems seem like they require a lot of spatial world knowledge, more than they require abstract reasoning. Shapes overlapping each other, containing each other, slicing up and reassembling pieces, denoising regular geometric shapes, you can call them "core knowledge" but to me it seems like they are more like "things that are intuitive to human visual processing". Would…

To parent: the spatial reasoning and blind person were great counterexamples. It still might be OK despite the blind exceptions if it showed general reasoning. To OP: I like your project goal. I think you should look at prior, reasoning engines that tried to build common sense. Cyc and OpenMind are examples. You also might find use for the list of AGI goals in Section 2 of this paper: https://arxiv.org/pdf/2308.04445…

CYC was an expert system, which is arguably what LLMs are.

A similar vintage GOFAI project that might do better on these, with a suitable visual front end, is SOAR - a general purpose problem solver.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#46

Post by bbor »

mark_l_watson wrote: post #32
pmayrgundter wrote: post #14

This claim that these tests are easy for humans seems dubious, and so I went looking a bit. Melanie Mitchell chimed in on Chollet's thread and posted their related test [ConceptARC]. In it they question the ease of Chollet's tests: "One limitation on ARC’s usefulness for AI research is that it might be too challenging. Many of the tasks in Chollet’s corpus are difficult even for humans, and the corpus as a whole migh…

I saw Melanie’s post and I am intrigued by an easier AGI suite. I would like some experimenting done by individuals like myself snd smaller organizations.

Are you working on (a book detailing) AGI also? It’s a lonely field but I have no doubt there are a sea of malcontent engineers across the world who saw the truth early on and are pushing solo for AGI. It’s going well for me, but I’m not sure whether to take that as “you’re great” or “it’s really that easy”, so was interested to see such a fellow brazen American on HN of all places.

Game on for the million, if so :). If not, apologies for distracting from the good fight for OSS/noncorp devs!

E: it occurred to me on the drive home how easily we (engineers) can fall into competitiveness, even when we’ve all read the thinkpieces about why an AI Race would/will be/is incredibly dangerous. Maybe not “game on”, perhaps… “god I hope it’s impossible but best of luck anyway to both of us”?

Re: ARC Prize – a $1M+ competition towards open AGI progress

#48

Post by goertzen »

salamo wrote: post #21

This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and ha…

I don’t know enough of biology or genetics or evolution, but surely the millions of years of training that is hardcoded into our genes and expressed in our biology had much larger “training” runs.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#49

Post by pants2 »

salamo wrote: post #21

This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and ha…

Humans, I would bet, could distinguish between two animals they've never seen based only on a loose or tangential description. I.e. "A dog hunts animals by tracking and chasing them long enough to exhaust their energy, but a cat is opportunistic and strikes using stealth and agility."

A human that has never seen a dog or a cat could probably determine which is which based on looking at the two animals and their adaptations. This would be an interesting test for AIs, but I'm not quite sure how one would formulate a eval for this.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#50

Post by HarHarVeryFunny »

abtinf wrote: post #20

> requires no world knowledge, no understanding of language This is treating “intelligence” like some abstract, platonic thing divorced from reality. Whatever else solving these puzzles is indicative of, it’s not intelligence.

Actually ARC fit's my definition of animal intelligence - "degree of ability to use prior experience to predict future outcomes".

Any useful definition of intelligence has to be totally general - to our brain experience is just patterns of neural activation. Our brain has no notion of certain inputs being from the the jungle and others from the blackboard or whatever.

Post reply on HN