Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC Prize – a $1M+ competition towards open AGI progress

arcprize.org

Post reply on HN
Print view View original

171–180 of 351 posts

Re: ARC Prize – a $1M+ competition towards open AGI progress

#171

Post by visarga »

8organicbits wrote: post #160
visarga wrote: post #148

Chollet's argument is that LLMs just imitate and recombine patterns. This might be true if you're looking at LLMs in isolation, but when they chat with people something different happens. The system made of humans+LLMs is an AGI. It is no longer just a parrot, it ingests new information, gets guidance, feedback and is basically embodied in a chat room with human and tools. This scales for 200M users and 1 billion ses…

> The system made of humans+LLMs is an AGI. Pay no attention to the man behind the curtain. This type of thinking would claim that mechanical turk is AGI, or perhaps that human+pen and paper is AGI. While they are great tools, that's not how I'd characterize them.

> Pay no attention to the man behind the curtain.

I could say the same for us, pay no attention to the other humans who are behind the curtain.

Humans in isolation are dumb, limited, and can get nowhere with understanding the world. Intelligence is mostly nurture over nature, the collective activity of society nurtures intelligence. It's smart because it learns from many diverse experiences and has a common language for sharing discoveries.

A human, even the smartest of us, can't solve cutting edge problems on demand, we're not that smart. But we can stumble on discoveries, especially in large numbers, and can share good ideas. We're smart by stumbling onto good ideas, and we can build upon these discoveries because we have a common language. Just a massive search program based on real world outcomes, that is what looks like general intelligence at societal level.

If you take the social aspect of intelligence into consideration then LLMs are judged in an inappropriate way, as stand alone agents. Of course they are limited, and we're almost as limited alone. The real locus of intelligence is the language-world system.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#172

Post by ein0p »

com2kid wrote: post #53
salamo wrote: post #21

This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and ha…

> humans do not need 10,000 examples to tell the difference between cats and dogs, I swear, not enough people have kids. Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. One thing kids do is they'll ask for confirmation of their guess. You'll be reading a book you've read 50 times before and the kid will stop you, point at a dog in the book, and ask "dog?" And there is a dev…

Not to mention that babies receive petabytes of visual input to go with other stimuli. It’s up for debate how sample efficient humans actually are in the first few years of their lives.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#173

Post by hackerlight »

paxys wrote: post #12

While I agree with the spirit of the competition, a $1M prize seems a little too low considering tens of billions of dollars have already been invested in the race to AGI, and we will see many times that put into the space in the coming years. The impact of AGI will be measured in trillions at minimum. So what you are ultimately rewarding isn't AGI research but fine tuning the newest public LLM release to best meet t…

The $1M ARC prize is advertising, just like being #1 on the huggingface leaderboard. It won't matter for end consumers, but for attracting the best talent it could be valuable.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#174

Post by lucianbr »

bigyikes wrote: post #109
kenjackson wrote: post #106

Earlier quoted context omitted.

How come? The pattern should work for any size grid.

You might be technically correct, but if you extend that logic, why not just make the grid 1x1 and select a single color? The grid size is part of the pattern in the same way that the colors are part of the pattern. It’s not just a color pattern, it’s a generalized mapping of input to output. In short: you need to resize the grid because that’s what the examples do.

What is even the meaning of "correct" in this case?

This makes me think of "math" problems requiring you to find the next number in a series. They give you 5 numbers, and ask for the 6th. When I can build a polynomial than can generate the first 5 and any 6th number. Any.

Sounds like the point of these exercises it to guess what the author had in mind, more than some universal intelligence test. Though of course the author thinks their own thoughts are the measure of universal intelligence. It's a tempting thing to believe.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#175

Post by lucianbr »

johndough wrote: post #134
bigyikes wrote: post #109

Earlier quoted context omitted.

You might be technically correct, but if you extend that logic, why not just make the grid 1x1 and select a single color? The grid size is part of the pattern in the same way that the colors are part of the pattern. It’s not just a color pattern, it’s a generalized mapping of input to output. In short: you need to resize the grid because that’s what the examples do.

> why not just make the grid 1x1 and select a single color? For two reasons: 1. The initially suggested grid size was 3x3. 2. Filling in a 3x3 grid is sufficient to show that you understood the pattern, but filling in a 1x1 (or even 2x2) grid is insufficient. Requiring the user fill in a larger grid is a waste of time. The existence of the grid size selector would still make sense in cases where a 2x2 grid would be s…

The fact that two intelligent beings are debating what the correct answer is shows that there is no fixed correct answer that proves "intelligence".

This is IQ tests all over again. Actually testing how alike you think to the author of the test.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#176

Post by gkbrk »

salamo wrote: post #34
pmayrgundter wrote: post #14

This claim that these tests are easy for humans seems dubious, and so I went looking a bit. Melanie Mitchell chimed in on Chollet's thread and posted their related test [ConceptARC]. In it they question the ease of Chollet's tests: "One limitation on ARC’s usefulness for AI research is that it might be too challenging. Many of the tasks in Chollet’s corpus are difficult even for humans, and the corpus as a whole migh…

They claim that the average score for humans is between 85% and 100%, so I think there's a disagreement on whether the test is actually too hard. Taking them at their word, if no existing model can score even half what the average human can, the test is certainly measuring some kind of significant difference. I guess there might be a disagreement of whether the problems in ARC are a representative sample of all of th…

The tasks are very easy for humans. Out of the 6 tasks assigned when I opened the web page, I got all of them correct on the first try.

Maybe if you run into some exceptionally difficult tasks it might not be 100%, but there's no way the challenge can be called unfair because it's too difficult for humans too.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#177

Post by gkbrk »

taneq wrote: post #82
bigyikes wrote: post #11

What is the fundamental difference between ARC and a standard IQ test? On the surface they seem similar in that they both involve deducing and generalizing visual patterns. Is there something special about these questions that makes them resistant to memorization? Or is it more just the fact that there are 100 secret tasks?

I’ve always found this kind of puzzle infuriating because it’s way underspecified. You’re not trying to find a pattern, you’re trying to guess what pattern the test writer would expect.

Most of the ARC tasks are intuitive and have one obvious answer. Both on IQ tests and the ARC challenge, people manage to guess what the test writer expects.

For an AI that's more useful anyway. If the task is specified completely non-ambiguously, you wouldn't need AI. But if it can correctly guess what you want from a limited number of obvious examples that's much more useful.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#178

Post by bubblyworld »

lucianbr wrote: post #162
bubblyworld wrote: post #141

Earlier quoted context omitted.

Why not? If the only thing that can solve problem X is AGI (e.g. humans), and something else comes along that solves it, then rationally that should be evidence that the something else is AGI right? Unless you have strong prior beliefs (like "computers can't be AGI") or something else that's problem specific ("these problems can be solved by these techniques which don't count as AGI"). So I guess that's my real quest…

That makes no sense at all. Any problem is initially only solvable by humans, until some technology is developed to solve it. Calculating a logarithm was at some point only doable by humans, and then digital computers came along. This would be in your view evidence that digital computers are AGI!? As in, an 8086 with some math code is AGI. We've had it for decades now, only nobody noticed :)

It's just Bayes theorem - there are basically two variables that control how strong the evidence is:

* How likely you think AGI is in general.

* How solvable you think the problem is, independently of what's solving it.

In the cases you've brought up that latter probability is very high, which means that they are extremely weak evidence that computers are AGI. So we agree!

In this case the latter probability seems to be quite low - attempts to solve it with computers have largely failed so far!

Re: ARC Prize – a $1M+ competition towards open AGI progress

#179

Post by bubblyworld »

educasean wrote: post #170
bubblyworld wrote: post #141

Earlier quoted context omitted.

Why not? If the only thing that can solve problem X is AGI (e.g. humans), and something else comes along that solves it, then rationally that should be evidence that the something else is AGI right? Unless you have strong prior beliefs (like "computers can't be AGI") or something else that's problem specific ("these problems can be solved by these techniques which don't count as AGI"). So I guess that's my real quest…

This flies directly in the face of technologies such as Deep Blue and AlphaGo. They excel in tiny domains previously thought to be the pinnacle of intelligence, and now they dominate humans. Are they AGI in your definition?

See my response to the other commenter. In these cases as well I would conclude it's very weak evidence of AGI, so I don't think we disagree.

Edit: I think maybe the disagreement here is about the nature of evidence. I think there can be evidence that something is AGI even if it isn't, in fact, AGI. You seem to believe that if there's any evidence that something is AGI, it must be AGI, I think?

Re: ARC Prize – a $1M+ competition towards open AGI progress

#180

Post by Animats »

> the only eval which measures AGI.

That's a stretch. This is a problem at which LLMs are bad. That does not imply it's a good measure of artificial general intelligence.

After working a few of the problems, I was wondering how many different transformation rules the problem generator has. Not very many, it seems. So the problem breaks down into extracting the set of transformation rules from the data, then applying them to new problems. The first part of that is hard. It's a feature extraction problem. The transformations seem to be applied rigidly, so once you have the transformation rules, and have selected the ones that work for all the input cases, application should be straightforward.

This seems to need explicit feature extraction, rather than the combined feature extraction and exploitation LLMs use. Has anyone extracted the rule set from the test cases yet?

Post reply on HN