Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC Prize – a $1M+ competition towards open AGI progress

arcprize.org

Post reply on HN
Print view View original

161–170 of 351 posts

Re: ARC Prize – a $1M+ competition towards open AGI progress

#161

Post by montag »

p1esk wrote: post #108

Is there a leaderboard for the no-restriction version of the competition? I want to see how gpt4 does on it.

Just quoting again from the guide:

3. DIRECT LLM PROMPTING In this method, contestants use a traditional LLM (like GPT-4) and rely on prompting techniques to solve ARC-AGI tasks. This was found to perform poorly, scoring "LLMs like Gemini or ChatGPT [don't work] because they're basically frozen at inference time. They're not actually learning anything." - François Chollet

Additionally, keep in mind that submissions to Kaggle will not have access to the internet. Using a 3rd-party, cloud-hosted LLM is not possible.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#162

Post by lucianbr »

bubblyworld wrote: post #141
mewpmewp2 wrote: post #118

Earlier quoted context omitted.

AGI should obviously be able to do them. But AI being able to do those 100 percent wouldn't be evidence of AGI however. It is a very narrow domain.

Why not? If the only thing that can solve problem X is AGI (e.g. humans), and something else comes along that solves it, then rationally that should be evidence that the something else is AGI right? Unless you have strong prior beliefs (like "computers can't be AGI") or something else that's problem specific ("these problems can be solved by these techniques which don't count as AGI"). So I guess that's my real quest…

That makes no sense at all. Any problem is initially only solvable by humans, until some technology is developed to solve it. Calculating a logarithm was at some point only doable by humans, and then digital computers came along. This would be in your view evidence that digital computers are AGI!? As in, an 8086 with some math code is AGI. We've had it for decades now, only nobody noticed :)

Re: ARC Prize – a $1M+ competition towards open AGI progress

#164

Post by interloxia »

educasean wrote: post #156
PheonixPharts wrote: post #77

Earlier quoted context omitted.

> Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. I have kids so I'm presuming I'm allowed to have an opinion here. This is ignoring the fact that babies are not just learning labels, they're learning the whole of language, motion planning, sensory processing, etc. Once they have the basics down concept acquisition time shrinks rapidly and kids can easily learn their new fa…

> kids can easily learn their new favorite animal in as little as a single example Until they encounter a similar animal and get confused, at which point you understand the implicit heuristic they were relying on. (Eg. They confused a dairy cow as a zebra, which means their heuristic was a black-and-white quadrupedal) Doesn't this seem remarkably close to how LLMs behave with one-shot or few-shot learning? I think th…

Not many zebras where I live but lots of little dogs. Small dogs were clearly cats for a long time no matter what I said. The training can take a while.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#165

Post by janalsncm »

geor9e wrote: post #147

I found them all extremely easy for a while, but then I couldn't figure out the rules of this one at all: e6de6e8f https://i.imgur.com/ExMFGqU.png

Each of the red shapes in the input are separated by black squares. Starting from the green block, rotate the red shapes 90 degrees and stack them downwards.

Thats the general pattern although my description wasn’t very good.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#166

Post by visarga »

Why doesn't Chollet just make a challenge that reads like "Solve cancer", surely there is no solution in any books.

If the AI is really AGI it could presumably do it. But not even the whole human society can do it in one go, it's a slow iterative process of ideation and validation. Even though this is a life and death matter, we can't simply solve it.

This is why AGI won't look like we expect, it will be a continuation of how societies solve problems. Intelligence of a single AI in isolation is not comparable to that of societies of agents with diverse real world interactions.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#167

Post by lucianbr »

andoando wrote: post #96
crazygringo wrote: post #19

Earlier quoted context omitted.

> Spatial relationships are just another type of logical relationship and we should expect AGI to be able to analyze relationships and generate algorithms on the fly to solve problems. Not really. By that reasoning, 5-dimensional spatial reasoning is "just another type of logical relationship" and yet humans mostly can't do that at all. It's clear that we have incredibly specialized capabilities for dealing with two-…

Literally every single thing you reason about is something happening in space-time.

Where exactly in space-time are complex numbers? Could you point me to 2+i for example?

How about some aliens in a SF book. When we reason about them, where are they exactly? Literally on the pages of the book?

How about a context-free grammar?

Re: ARC Prize – a $1M+ competition towards open AGI progress

#168

Post by llm_trw »

com2kid wrote: post #53
salamo wrote: post #21

This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and ha…

> humans do not need 10,000 examples to tell the difference between cats and dogs, I swear, not enough people have kids. Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. One thing kids do is they'll ask for confirmation of their guess. You'll be reading a book you've read 50 times before and the kid will stop you, point at a dog in the book, and ask "dog?" And there is a dev…

Babies, unlike machine learning models, aren't placed in limbo when they aren't running back propagation.

Babies need few examples for complex tasks because they get constant infinitely complex examples on tasks which are used for transfer learning.

Current models take a nuclear reactors worth of power to run back prop on top of a small countries GDP worth of hardware.

They are _not_ going to generalize to AGI because we can't afford to run them.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#169

Post by llm_trw »

theptip wrote: post #100
salamo wrote: post #21

This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and ha…

> humans do not need 10,000 examples to tell the difference between cats and dogs The optimization process that trained the human brain is called evolution, and it took a lot more than 10,000 examples to produce a system that can differentiate cats vs dogs. Put differently, an LLM is pre-trained with very light priors, starting almost from scratch, whereas a human brain is pre-loaded with extremely strong priors.

>The optimization process that trained the human brain is called evolution

A human brain that doesn't get visual stimulus at the critical age between 0 and 3 years old will never be able to tell the difference between a cat and a dog because it will be forevermore blind.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#170

Post by educasean »

bubblyworld wrote: post #141
mewpmewp2 wrote: post #118

Earlier quoted context omitted.

AGI should obviously be able to do them. But AI being able to do those 100 percent wouldn't be evidence of AGI however. It is a very narrow domain.

Why not? If the only thing that can solve problem X is AGI (e.g. humans), and something else comes along that solves it, then rationally that should be evidence that the something else is AGI right? Unless you have strong prior beliefs (like "computers can't be AGI") or something else that's problem specific ("these problems can be solved by these techniques which don't count as AGI"). So I guess that's my real quest…

This flies directly in the face of technologies such as Deep Blue and AlphaGo. They excel in tiny domains previously thought to be the pinnacle of intelligence, and now they dominate humans. Are they AGI in your definition?
Post reply on HN