Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC Prize – a $1M+ competition towards open AGI progress

arcprize.org

Post reply on HN
Print view View original

21–30 of 351 posts

Re: ARC Prize – a $1M+ competition towards open AGI progress

#21

Post by salamo »

This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and hard to synthesize.

If I can make one criticism/observation of the tests, it seems that most of them reason about perfect information in a game-theoretic sense. However, many if not most of the more challenging problems we encounter involve hidden information. Poker and negotiations are examples of problem solving in imperfect information scenarios. Smoothly navigating social situations also requires a related problem of working with hidden information.

One of the really interesting things we humans are able to do is to take the rules of a game and generate strategies. While we do have some algorithms which can "teach themselves" e.g. to play go or chess, those same self-play algorithms don't work on hidden information games. One of the really interesting capabilities of any generally-intelligent system would be synthesizing a general problem solver for those kinds of situations as well.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#22

Post by gkamradt »

lxe wrote: post #16

I've never done these before, or Kaggle competitions in general. Any recommendations before I dive in? I have prety much zero lowe-level ML experience, but a good amount of practical software eng behind me.

We put a bunch of detail to get started on the guide https://arcprize.org/guide

Happy to answer any questions you have along the way

(I'm helping run ARC Prize)

Re: ARC Prize – a $1M+ competition towards open AGI progress

#23

Post by gkamradt »

david_shi wrote: post #18

What is the fastest way to get up to speed with techniques that led to the current SOTA?

Check out the SOTA resources on the guide

https://arcprize.org/guide

Happy to answer any questions you have along the way

(I'm helping run ARC Prize)

Re: ARC Prize – a $1M+ competition towards open AGI progress

#25

Post by abtinf »

abtinf wrote: post #20

> requires no world knowledge, no understanding of language This is treating “intelligence” like some abstract, platonic thing divorced from reality. Whatever else solving these puzzles is indicative of, it’s not intelligence.

From the abstract of the “ On the Measure of Intelligence” paper:

> We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience.

I’m afraid that definition forecloses the possibility of AGI. The immediate basic question is: why build skills at all?

Re: ARC Prize – a $1M+ competition towards open AGI progress

#26

Post by nojvek »

I love the ARC challenge. It's hard to beat by memorization. There aren't enough examlples, so one has to train on a large dataset elsewhere and then train on ARC to generalize and figure out which rules are most applicable.

I did a few human examples by hand, but gotta do more of them to start seeing patterns.

Human visual and auditory system is impressive. Most animals see/hear and plan from that without having much language. Physical intelligence is the biggest leg up when it comes to evolution optimizing for survival.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#27

Post by ks2048 »

paxys wrote: post #12

While I agree with the spirit of the competition, a $1M prize seems a little too low considering tens of billions of dollars have already been invested in the race to AGI, and we will see many times that put into the space in the coming years. The impact of AGI will be measured in trillions at minimum. So what you are ultimately rewarding isn't AGI research but fine tuning the newest public LLM release to best meet t…

The submissions can't use the internet. And I imagine can't be too huge - so you can't use "newest public LLMs" on this task.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#28

Post by mikeknoop »

paxys wrote: post #12

While I agree with the spirit of the competition, a $1M prize seems a little too low considering tens of billions of dollars have already been invested in the race to AGI, and we will see many times that put into the space in the coming years. The impact of AGI will be measured in trillions at minimum. So what you are ultimately rewarding isn't AGI research but fine tuning the newest public LLM release to best meet t…

I agree, $1M is ~trivial in AI. The primary goal with the prize is to raise public awareness about how close (or far today) we are from AGI: https://arcprize.org/leaderboard and we hope that understanding will shift more would-be AI researchers to working new ideas

Re: ARC Prize – a $1M+ competition towards open AGI progress

#29

Post by Phil_Latio »

abtinf wrote: post #20

> requires no world knowledge, no understanding of language This is treating “intelligence” like some abstract, platonic thing divorced from reality. Whatever else solving these puzzles is indicative of, it’s not intelligence.

Why does an AGI need to have any knowledge about our reality? The principle behind an AGI should work just as well on a made up world where those puzzles play a part in.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#30

Post by mikeknoop »

ks2048 wrote: post #27
paxys wrote: post #12

While I agree with the spirit of the competition, a $1M prize seems a little too low considering tens of billions of dollars have already been invested in the race to AGI, and we will see many times that put into the space in the coming years. The impact of AGI will be measured in trillions at minimum. So what you are ultimately rewarding isn't AGI research but fine tuning the newest public LLM release to best meet t…

The submissions can't use the internet. And I imagine can't be too huge - so you can't use "newest public LLMs" on this task.

That is correct for ARC Prize: limited Kaggle compute (to target efficiency) and no internet (to reduce cheating).

We are also trialing a secondary leaderboard called ARC-AGI-Pub that imposes no limits or constraints. Not part of the prize today but could be in the future: https://arcprize.org/leaderboard

Post reply on HN