I can see many problems can be solved with modern symbolic approaches like theorem provers, dependent types, pattern matching etc. But I will have to dive in to actually confirm it.
ARC Prize – a $1M+ competition towards open AGI progress
131–140 of 351 posts
Re: ARC Prize – a $1M+ competition towards open AGI progress
#132Are we allowed to combine multiple tools including gpt-4 to solve this? E.g. a script that does image processing, passes the results to gpt, where gpt can invoke further runs of scripts using other tools?
> submissions to Kaggle will not have access to the internet. Using a 3rd-party, cloud-hosted LLM is not possible.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#133I'm curious, if it turns out that a simple rule-based algorithm exists, specifically tailored to solve (only!) ARC style problems, without generalization, would that still qualify for the reward?
I don't think that's breaking any rules, and in fact it would help to expose a whole class of weaknesses in the test.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#134Earlier quoted context omitted.
How come? The pattern should work for any size grid.
You might be technically correct, but if you extend that logic, why not just make the grid 1x1 and select a single color? The grid size is part of the pattern in the same way that the colors are part of the pattern. It’s not just a color pattern, it’s a generalized mapping of input to output. In short: you need to resize the grid because that’s what the examples do.
> why not just make the grid 1x1 and select a single color?
For two reasons:
1. The initially suggested grid size was 3x3.
2. Filling in a 3x3 grid is sufficient to show that you understood the pattern, but filling in a 1x1 (or even 2x2) grid is insufficient.
Requiring the user fill in a larger grid is a waste of time. The existence of the grid size selector would still make sense in cases where a 2x2 grid would be sufficient to show the solution, so it is not obvious at all that a 6x6 grid should be chosen.
> The grid size is part of the pattern in the same way that the colors are part of the pattern.
To understand a pattern, you have to see at least two valid inputs and corresponding outputs. For the first example, a valid example for the expected output grid size is missing.
I arrived at the "correct" conclusion eventually, but the only indicator was that the reading direction for the UI was absolutely ridiculous ( https://i.imgur.com/CuQ2z2N.png ), suggesting that the authors did not think this through properly, so the solution had to be weird as well.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#135This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and ha…
> humans do not need 10,000 examples to tell the difference between cats and dogs, I swear, not enough people have kids. Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. One thing kids do is they'll ask for confirmation of their guess. You'll be reading a book you've read 50 times before and the kid will stop you, point at a dog in the book, and ask "dog?" And there is a dev…
That’s all true, yet my 2.5 year old sometimes one-shots specific information. I told my daughter that woodpeckers eat bugs out of trees after doing what you said and asking “what’s that noise?” for the fifth time in a few minutes when we heard some this spring. She brought it up again at least a week later, randomly. Developing brains are amazing.
She also saw an eagle this spring out the car window and said “an eagle! …no, it’s a bird,” so I guess she’s still working on those image classifications ;)
Re: ARC Prize – a $1M+ competition towards open AGI progress
#136I'm Simon Strandgaard and I participated in ARCathon 2022 (solved 3 tasks) and ARCathon 2023 (solved 8 tasks). I'm collecting data for how humans are solving ARC tasks, and so far collected 4100 interaction histories ( https://github.com/neoneye/ARC-Interactive-History-Dataset ). Besides ARC-AGI, there are other ARC like datasets, these can be tried in my editor ( https://neoneye.github.io/arc/ ). I have made some vi…
"Here is a challenge, designed to be unsolvable or so. We'll give you a bazillion dollars if you complete the challenge, and, in the meantime, we will use your attempts to train an as AI that will be worth the cost!!"
Did you even try the puzzles? They’re not particularly “unsolvable”.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#137This is interesting. I've been looking at the data today and made a helper to quickly view the ARC dataset: https://kts.github.io/arc-viewer/
So you can view 100 per page instead of clicking through one-by-one: https://kts.github.io/arc-viewer/page1/
Re: ARC Prize – a $1M+ competition towards open AGI progress
#138François Chollet's original paper is incredibly insightful and I'm consistently shocked more people don't talk about it. Some parts are quite technical but at a high level it is the best answer to "what do we mean by general intelligence?" that I've yet seen. Defining intelligence as an efficiency of learning , after accounting for any explicit or implicit priors about the world, makes it much easier to understand wh…
Do you remember the title/where to find it?
Re: ARC Prize – a $1M+ competition towards open AGI progress
#139Are we allowed to combine multiple tools including gpt-4 to solve this? E.g. a script that does image processing, passes the results to gpt, where gpt can invoke further runs of scripts using other tools?
> submissions to Kaggle will not have access to the internet. Using a 3rd-party, cloud-hosted LLM is not possible. https://arcprize.org/guide
This largely takes away any odds at solving this. You definitely can't reproduce that under a million dollars.
I have some ideas I want to try, I might still though. But all of it would require external tools.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#140I'm Simon Strandgaard and I participated in ARCathon 2022 (solved 3 tasks) and ARCathon 2023 (solved 8 tasks). I'm collecting data for how humans are solving ARC tasks, and so far collected 4100 interaction histories ( https://github.com/neoneye/ARC-Interactive-History-Dataset ). Besides ARC-AGI, there are other ARC like datasets, these can be tried in my editor ( https://neoneye.github.io/arc/ ). I have made some vi…
"Here is a challenge, designed to be unsolvable or so. We'll give you a bazillion dollars if you complete the challenge, and, in the meantime, we will use your attempts to train an as AI that will be worth the cost!!"
No, you missed the point. The striking thing about ARC is the puzzles are super easy, for humans. The average person solves 85% of the tasks, but the worlds best LLMs are only solving 5%. The challenge is to simply make an AI score as well as the average human.