Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC Prize – a $1M+ competition towards open AGI progress

arcprize.org

Post reply on HN
Print view View original

231–240 of 351 posts

Re: ARC Prize – a $1M+ competition towards open AGI progress

#231

Post by ben_w »

TeMPOraL wrote: post #214
_flux wrote: post #195

Earlier quoted context omitted.

Are you saying it's not fair for LLMs, because of the way they are taught is different? The difference is that we don't know better methods for them, but we do know of better methods for people.

I think they're saying that it's silly to claim humans learn with less data than LLMs, when humans are ingesting a continuous video, audio, olfactory and tactile data stream for 16+ hours a day, every day. It takes at least 4 years for a human children to be in any way comparable in performance to GPT-4 on any task both of them could be tested on; do people really believe GPT-4 was trained with more data than a 4 yea…

> do people really believe GPT-4 was trained with more data than a 4 year old?

I think it was; the guesstimate I've seen is GPT-4 was trained on 13e12 tokens, that over 4 years is 8.9e9/day or about 1e5/s.

Then it's a question of how many bits per token — my expectation is 100k/s is more than the number of token-equivalents we experience, even though it's much less than the bitrate even of just our ears let alone our eyes.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#232

Post by HarHarVeryFunny »

I have two questions:

1) Who is providing the prize money, and if it is yourself and Francois personally, then what is your motivation ?

2) Do you think it's possible to create a word-based, non-spatial (not crosswords or sudoku, etc) ARC test that requires similar run-time exploration and combination of skills (i.e. is not amenable to a hoard of narrow skills)?

Re: ARC Prize – a $1M+ competition towards open AGI progress

#233

Post by skywhopper »

“Given the success and proven economic utility of LLMs over the past 4 years, the above may seem like extraordinary claims. Strong claims require strong evidence.”

Speaking of extraordinary claims. What evidence is there that LLMs have “proven economic utility”? They’ve drawn a ludicrous amount of investment thanks to claims of future economic utility, but I’ve yet to see any evidence of it.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#234

Post by mewpmewp2 »

visarga wrote: post #166

Why doesn't Chollet just make a challenge that reads like "Solve cancer", surely there is no solution in any books. If the AI is really AGI it could presumably do it. But not even the whole human society can do it in one go, it's a slow iterative process of ideation and validation. Even though this is a life and death matter, we can't simply solve it. This is why AGI won't look like we expect, it will be a continuati…

AGI can't necessarily solve cancer. Perhaps ASI could (but maybe not), but AGI can only do what the most talented people can do in their areas of expertise or actions. So since people haven't solved cancer, that's not a requirement to be AGI.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#235

Post by HarHarVeryFunny »

TeMPOraL wrote: post #212
interloxia wrote: post #164

Earlier quoted context omitted.

Not many zebras where I live but lots of little dogs. Small dogs were clearly cats for a long time no matter what I said. The training can take a while.

This. My 2.5 y.o. still argues with me that a small dog she just saw in the park is a "cat". That's in contrast to her older sister, who at 5 is... begrudgingly accepting that I might be right about it after the third time I correct her.

The thing is that the labels "cat" and "dog" reflect a choice in most languages to name animals based on species, which manifests in certain physical/behavioral attributes. Children need to learn by observation/teaching and generalization that these are the characteristics they need to use to conform to our chosen labelling/distinction, and that other things such as size/color/speed are irrelevant.

Of course it didn't have to be this way - in a different language animals might be named based on size or abilities/behavior, etc.

So, your daughter wanting to label a cat-sized dog as a cat is just a reflection of her not having aligned her generalization of what you are talking about when you say "cat" vs "dog" with her own.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#236

Post by smusamashah »

Tepix wrote: post #205
smusamashah wrote: post #158

Earlier quoted context omitted.

My kid is about 3 and has been slow on language development. He can barely speak a few short sentences now. Learning names of things and concepts made a big difference for him and that's a fascinating watch and realization. This reminds of the story of Adam learning names, or how some languages can express a lot more in fewer words. And it makes sense that LLMs look intelligent to us. My kid loves repeating the names…

> I think we learn fast because of stereo (3d) vision. I think stereo vision is not that important if you can move around and get spatial clues that way also.

Every animal/insect I can think of has more than 1 eye. Some has lot more than 2 eyes. It has to be that important.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#237

Post by [deleted] »

janalsncm wrote: post #165
geor9e wrote: post #147

I found them all extremely easy for a while, but then I couldn't figure out the rules of this one at all: e6de6e8f https://i.imgur.com/ExMFGqU.png

Each of the red shapes in the input are separated by black squares. Starting from the green block, rotate the red shapes 90 degrees and stack them downwards. Thats the general pattern although my description wasn’t very good.

[deleted]

Re: ARC Prize – a $1M+ competition towards open AGI progress

#238

Post by mkl »

I did https://arcprize.org/play?task=05a7bcf2 correctly, but one of the examples doesn't match the rule I used. Are the examples supposed to contain mistakes/noise? Did I find a bug? Did I get the rule wrong?

Here's how I understand the rule: yellow blobs turn green then spew out yellow strips towards the blue line, and the width of the strips is the number of squares the green blobs take up along the blue line. The yellow strips turn blue when they hit the blue line, then continue until they hit red, then they push the red blocks all the way to the other side, without changing the arrangement of the red blocks that were in the way of the strip.

The first example violates the last bit. The red blocks in the way of the rightmost strip start as

  R
  R R
  R R R
but get turned into

  R R
  R R
  R R R
Every other strip matches my rule.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#239

Post by HarHarVeryFunny »

mewpmewp2 wrote: post #118
HarHarVeryFunny wrote: post #40

Earlier quoted context omitted.

I just did the first 5 of the "public eval set" without having looked at the "public training set", and found them easy enough. If we're defining AGI as at least human level, then the AGI should also be able to do these without seeing any more examples. I don't think there's any rules about what knowledge/experience you build into your solution.

AGI should obviously be able to do them. But AI being able to do those 100 percent wouldn't be evidence of AGI however. It is a very narrow domain.

Yes, a narrow domain, but the core capability it is testing for (explorative combination/application of learned patterns and skills) is a general one that in a meaningful AGI would be available across domains.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#240

Post by nadam »

I love this, this is super interesting, but my intuition based on looking at a dozen examples is that the problem is hard, but easy enough that if this problem becomes popular, near-human level results will appear in a year or less, and AGI will not be reached. The problem seems to be finding a generic enough transformation description language with the appropriate operators. And then heuristics to find a very short program (in the information theoretical sense) in this language that produces all the examples for a problem. I would be very surprised if we would not increase the 34% result soon significantly, and I would be surprised if this could be transferred to general intelligence, at least when I think of the topics where I use AI today and where it falls short yet. Basically my intuition is that this will be yet another 'Chess' or 'Go'-like problem in AI. But still a worthwhile research topic, absolutely: the value that could come out of this is well worth the 1M dollars.
Post reply on HN