Live data from Hacker News

ARC Prize – a $1M+ competition towards open AGI progress

arcprize.org

311–320 of 351 posts

Re: ARC Prize – a $1M+ competition towards open AGI progress

#311

Earlier quoted context omitted.

I'm sure you saw over 1B images of cats though, assuming 24 images per second from vision.

> I'm sure you saw over 1B images of cats though, assuming 24 images per second from vision. The AI models aren't seeing the same image 1B times.

Neither are you, during those 10 000 hours most of the time you aren't absolutely still.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#312
post #223

Earlier quoted context omitted.

> If the only thing that can solve problem X is AGI (e.g. humans), and something else comes along that solves it, then rationally that should be evidence that the something else is AGI right? No. Because there might undiscovered ways to solve these problems that no one claims is AGI. The definition of AGI is notoriously fuzzy, but non-the-less if there was a 10 line python program (with no external dependencies or da…

I think I agree with you, but consider these two cases: 1. Only humans are known to have solved problem X, and we've spent no time looking for alternative solutions. 2. Only humans are known to have solved problem X, and we've spent hundreds of thousands of hours looking for alternative solutions and failed. Now suppose something solves the problem. I feel like in case 2 we are justified in saying there's evidence th…

Maybe (2). But it took ~50 years work to build systems that can beat people at poker and I don't think people argue poker bots are AGI.

To be clear, I think we have AGI (LLMs with tool use are generalized enough) and we are currently finding edge cases that they fail at.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#313
post #312

Earlier quoted context omitted.

I think I agree with you, but consider these two cases: 1. Only humans are known to have solved problem X, and we've spent no time looking for alternative solutions. 2. Only humans are known to have solved problem X, and we've spent hundreds of thousands of hours looking for alternative solutions and failed. Now suppose something solves the problem. I feel like in case 2 we are justified in saying there's evidence th…

Maybe (2). But it took ~50 years work to build systems that can beat people at poker and I don't think people argue poker bots are AGI. To be clear, I think we have AGI (LLMs with tool use are generalized enough) and we are currently finding edge cases that they fail at.

> I think we have AGI

That seems a pretty extreme position!

What's your definition of AGI ?

Re: ARC Prize – a $1M+ competition towards open AGI progress

#314
post #27
post #12

While I agree with the spirit of the competition, a $1M prize seems a little too low considering tens of billions of dollars have already been invested in the race to AGI, and we will see many times that put into the space in the coming years. The impact of AGI will be measured in trillions at minimum. So what you are ultimately rewarding isn't AGI research but fine tuning the newest public LLM release to best meet t…

The submissions can't use the internet. And I imagine can't be too huge - so you can't use "newest public LLMs" on this task.

Using the internet would leak the test data, a big problem with ML benchmarks, and also allow communication with humans during the test.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#315
post #12

While I agree with the spirit of the competition, a $1M prize seems a little too low considering tens of billions of dollars have already been invested in the race to AGI, and we will see many times that put into the space in the coming years. The impact of AGI will be measured in trillions at minimum. So what you are ultimately rewarding isn't AGI research but fine tuning the newest public LLM release to best meet t…

They thought of that and so have yearly $100,000 in yearly prizes for the best results as well, so things can build up towards someone winning the $1 million over time: the yearly prizes require you to publish the techniques.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#316
post #312

Earlier quoted context omitted.

Maybe (2). But it took ~50 years work to build systems that can beat people at poker and I don't think people argue poker bots are AGI. To be clear, I think we have AGI (LLMs with tool use are generalized enough) and we are currently finding edge cases that they fail at.

> I think we have AGI That seems a pretty extreme position! What's your definition of AGI ?

> That seems a pretty extreme position!

Not really.

Jeremy Howard has said the same thing for example.

> What's your definition of AGI ?

Things that we consider intelligent when humans do them.

Basically we had all these definitions of AGI that we have surpassed (Turing test etc). Now we are finding more edge cases where we go "ahh... it can't do this so therefore it isn't intelligent".

But the issue with that is that lots of humans can't do them either.

I think the ARC challenge is valid. But I'd also point out that there are substantial numbers of people who won't be able to solve them either (blind people for example, as well as people who aren't good at puzzles). We make excuses there ("oh we can explain it to a blind person" or for many physical problems things like "Oh Stephen Hawking couldn't solve this but that is an exception") but we don't allow the same excuses for machine intelligence.

I don't think the boundary of AGI is a hard line, but if you went back 10 years and took what we had now and showed it to them I think people would be "Oh wow you have AI!".

Re: ARC Prize – a $1M+ competition towards open AGI progress

#317

Earlier quoted context omitted.

I'd say that was more like a single instance, one interaction with a thing.

One interaction that captures a multidimensional, multisensory set of perceptions. In an ML training set, say for visual recognition, this would consist at least of hundreds of images from many angles, in different poses and varied lighting.

I don't think its analogous, I don't think we see a cat and our brain have it frame by frame adjust our synaptic weights (or whatever brains do). The whole premise of natural brains being able to learn by static images or disjointed modalities is a very clunky reductionist engineered approach we have taken.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#318

Earlier quoted context omitted.

One interaction that captures a multidimensional, multisensory set of perceptions. In an ML training set, say for visual recognition, this would consist at least of hundreds of images from many angles, in different poses and varied lighting.

I don't think its analogous, I don't think we see a cat and our brain have it frame by frame adjust our synaptic weights (or whatever brains do). The whole premise of natural brains being able to learn by static images or disjointed modalities is a very clunky reductionist engineered approach we have taken.

> I don't think we see a cat and our brain have it frame by frame adjust our synaptic weights (or whatever brains do)

I think that "whatever we do" is doing a lot of heavy lifting here. Some of those "whatevers" will be isomorphic to a frame-level analysis that pulls out structural commonalities, or close enough that it's not a clunky reductionist analogy.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#319

Earlier quoted context omitted.

> why not just make the grid 1x1 and select a single color? For two reasons: 1. The initially suggested grid size was 3x3. 2. Filling in a 3x3 grid is sufficient to show that you understood the pattern, but filling in a 1x1 (or even 2x2) grid is insufficient. Requiring the user fill in a larger grid is a waste of time. The existence of the grid size selector would still make sense in cases where a 2x2 grid would be s…

The fact that two intelligent beings are debating what the correct answer is shows that there is no fixed correct answer that proves "intelligence". This is IQ tests all over again. Actually testing how alike you think to the author of the test.

Honestly I’d disagree. I was a bit confused at first but moment I realized I could resize the grid, the answer strikes me as obvious and clear. Yes, in some theoretic sense you can argue a 3 x 3 grid answer is fine, but shows this to 100 different humans and majority would agree that resizing the grid is the obvious and more natural solution.

Re: ARC Prize – a $1M+ competition towards open AGI progress

#320

Earlier quoted context omitted.

> Memorization is literally how you learned arithmetic, multiplication tables and fractions I understood how to do arithmetic for numbers with multiple digits before I was taught a "procedure". Also, I am not even sure what you mean by "memorization is how you learned fractions". What is there to memorize?

> I understood how to do arithmetic for numbers with multiple digits before I was taught a "procedure" What did you understand, exactly? You understood how to "count" using "numbers" that you also memorized? You intuitively understood that addition was counting up and subtraction was counting down, or did you memorize those words and what they meant in reference to counting? > Also, I am not even sure what you mean b…

Fractions is exactly an area of mathematics where I learned by understanding the concept and how it was represented and then would use that understanding to re-reason the procedures I had a hard time remembering.

I do have the single digit multiplication table memorized now, but there was a long time where that table had gaps and I would use my understanding of how numbers worked to to calculate the result rather than remembering it. That same process still occurs for double digit number.

Mathematics education, especially historically, has indeed leaned pretty heavily on memorization. That does mean thats the only way to learn math, or even a particularly good one. I personally think over reliance on memorization is part of why so many people think they hate math.

Post reply on HN