Live data from Hacker News

ARC Prize – a $1M+ competition towards open AGI progress

arcprize.org

51–60 of 351 posts

Re: ARC Prize – a $1M+ competition towards open AGI progress

#51

This claim that these tests are easy for humans seems dubious, and so I went looking a bit. Melanie Mitchell chimed in on Chollet's thread and posted their related test [ConceptARC]. In it they question the ease of Chollet's tests: "One limitation on ARC’s usefulness for AI research is that it might be too challenging. Many of the tasks in Chollet’s corpus are difficult even for humans, and the corpus as a whole migh…

I just tried the first puzzle and I can't get it right. I think my solution makes logical sense and I explain why the patterns are consistent with the input, but it says its wrong. I'm either a lot dumber than I thought or they need to do a better job of vetting their tests.

(You can direct link to a task like this: https://arcprize.org/play?task=009d5c81 in case you want to share!)

Re: ARC Prize – a $1M+ competition towards open AGI progress

#52

This claim that these tests are easy for humans seems dubious, and so I went looking a bit. Melanie Mitchell chimed in on Chollet's thread and posted their related test [ConceptARC]. In it they question the ease of Chollet's tests: "One limitation on ARC’s usefulness for AI research is that it might be too challenging. Many of the tasks in Chollet’s corpus are difficult even for humans, and the corpus as a whole migh…

I just tried the first puzzle and I can't get it right. I think my solution makes logical sense and I explain why the patterns are consistent with the input, but it says its wrong. I'm either a lot dumber than I thought or they need to do a better job of vetting their tests.

It's pretty easy, just follow the second example with the colors from the test input. (if it's the same puzzle 00576224 for you too)

Re: ARC Prize – a $1M+ competition towards open AGI progress

#53
post #21

This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and ha…

> humans do not need 10,000 examples to tell the difference between cats and dogs,

I swear, not enough people have kids.

Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands.

One thing kids do is they'll ask for confirmation of their guess. You'll be reading a book you've read 50 times before and the kid will stop you, point at a dog in the book, and ask "dog?"

And there is a development phase where this happens a lot.

Also kids can get mad if they are told an object doesn't match up to the expected label, e.g. my son gets really mad if someone calls something by the wrong color.

Another thing toddlers like to do is play silly labeling games, which is different than calling something the wrong name on accident, instead this is done on purpose for fun. e.g. you point to a fish and say "isn't that a lovely llama!" at which point the kid will fall down giggling at how silly you are being.

The human brain develops really slowly[1], and a sense of linear time encoding doesn't really exist for quite awhile. (Even at 3, everything is either yesterday, today, or tomorrow) so who the hell knows how things are being processed, but what we do know is that kids gather information through a bunch of senses, that are operating at an absurd data collection rate 12-14 hours a day, with another 10-12 hours of downtime to process the information.

[1] Watch a baby discover they have a right foot. Then a few days later figure out they also have a left foot. Watch kids who are learning to stand develop a sense of "up above me" after they bonk their heads a few time on a table bottom. Kids only learn "fast" in the sense that they have nothing else to do for years on end.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#54
François Chollet's original paper is incredibly insightful and I'm consistently shocked more people don't talk about it. Some parts are quite technical but at a high level it is the best answer to "what do we mean by general intelligence?" that I've yet seen.

Defining intelligence as an efficiency of learning, after accounting for any explicit or implicit priors about the world, makes it much easier to understand why human intelligence is so impressive.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#55
post #12

While I agree with the spirit of the competition, a $1M prize seems a little too low considering tens of billions of dollars have already been invested in the race to AGI, and we will see many times that put into the space in the coming years. The impact of AGI will be measured in trillions at minimum. So what you are ultimately rewarding isn't AGI research but fine tuning the newest public LLM release to best meet t…

That was my initial reaction too.

"Endow circuitry with consciousness and win a gift certificate for Denny's (may not be used in conjunction with other specials)"

Re: ARC Prize – a $1M+ competition towards open AGI progress

#56
post #16

I've never done these before, or Kaggle competitions in general. Any recommendations before I dive in? I have prety much zero lowe-level ML experience, but a good amount of practical software eng behind me.

We put a bunch of detail to get started on the guide https://arcprize.org/guide Happy to answer any questions you have along the way (I'm helping run ARC Prize)

I don't see where this helps @Ixe with getting started (me being in a similar state like him).

Re: ARC Prize – a $1M+ competition towards open AGI progress

#58
post #38

Earlier quoted context omitted.

Why does an AGI need to have any knowledge about our reality? The principle behind an AGI should work just as well on a made up world where those puzzles play a part in.

A concept that doesn’t relate to an aspect of reality, either directly or abstracted from basic concepts that directly relate, is meaningless and arbitrary. There is no way for intelligence to grasp it, let alone do something with it. To put it another way, a thing that solves puzzles without an understanding of reality is a calculator. When it solves a problem, it is the creator’s intelligence solving the problem, n…

I agree that the puzzles alone are not enough, that's why I wrote "in a made up world where those puzzles play a part in".

We are not looking for a superhuman, but for the (or a) mechanism of intelligence, which we can then transfer into a superhuman (into the real world). But the mechanism itself should work in an artifically made and very constrained world too.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#59
post #20

> requires no world knowledge, no understanding of language This is treating “intelligence” like some abstract, platonic thing divorced from reality. Whatever else solving these puzzles is indicative of, it’s not intelligence.

This argument is not very strong: is "physical strength" some abstract, platonic thing divorced from reality? Does a person's bench press, squat, deadlift, and overhead press capabilities have nothing to do with strength?

Or instead, is there some underlying latent capability we call 'strength,' that is correlated with performance in a broad but constrained range of real-world tasks that humans encounter and solve, whose value is something we'd like to assess and, ideally, build machines that can surpass?

Re: ARC Prize – a $1M+ competition towards open AGI progress

#60
post #52

Earlier quoted context omitted.

I just tried the first puzzle and I can't get it right. I think my solution makes logical sense and I explain why the patterns are consistent with the input, but it says its wrong. I'm either a lot dumber than I thought or they need to do a better job of vetting their tests.

It's pretty easy, just follow the second example with the colors from the test input. (if it's the same puzzle 00576224 for you too)

https://arcprize.org/play?task=00576224 Yes the same puzzle.

And I followed the second example. This was my solution:

GRG

OBO

RGR

B is the cyan like blue color. My solution looks right, but it says it’s wrong.

Post reply on HN