201–210 of 351 posts
This is interesting. I've been looking at the data today and made a helper to quickly view the ARC dataset: https://kts.github.io/arc-viewer/ So you can view 100 per page instead of clicking through one-by-one: https://kts.github.io/arc-viewer/page1/
Idea for a metric: - Number of pixels that stays the same between input/output. - Histogram changes.
Earlier quoted context omitted.
illusion that these models are learning like children do, when in reality they are not even entirely capable of learning novel concepts Now imagine how much would your kid learn if the only input he ever received was a sequence of words?
Are you saying it's not fair for LLMs, because of the way they are taught is different? The difference is that we don't know better methods for them, but we do know of better methods for people.
Earlier quoted context omitted.
I think the ARC-AGI tasks was manually drawn with an early version of fchollet's editor. Recently Michael Hodel has reverse engineered 400 of the tasks, so more tasks can be generated. Interestingly it can generate python programs that solves the tasks too. https://github.com/michaelhodel/re-arc
What do you mean it can 'generate python programs that solve the tasks'? I can't find any mention of that. I only see hand-coded solutions.
On puzzle #23 (id: 11e1fe23), I'm sure there's more than one possible valid answer from the examples given. You can't tell if the expected distance is from the gray square or from the RGB squares.
There are many examples where the test is slightly OOD (out of distribution), so the solver will have to generalize.
Earlier quoted context omitted.
> humans do not need 10,000 examples to tell the difference between cats and dogs, I swear, not enough people have kids. Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. One thing kids do is they'll ask for confirmation of their guess. You'll be reading a book you've read 50 times before and the kid will stop you, point at a dog in the book, and ask "dog?" And there is a dev…
My kid is about 3 and has been slow on language development. He can barely speak a few short sentences now. Learning names of things and concepts made a big difference for him and that's a fascinating watch and realization. This reminds of the story of Adam learning names, or how some languages can express a lot more in fewer words. And it makes sense that LLMs look intelligent to us. My kid loves repeating the names…
I think stereo vision is not that important if you can move around and get spatial clues that way also.
Earlier quoted context omitted.
> humans do not need 10,000 examples to tell the difference between cats and dogs, I swear, not enough people have kids. Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. One thing kids do is they'll ask for confirmation of their guess. You'll be reading a book you've read 50 times before and the kid will stop you, point at a dog in the book, and ask "dog?" And there is a dev…
Not to mention that babies receive petabytes of visual input to go with other stimuli. It’s up for debate how sample efficient humans actually are in the first few years of their lives.
Thank you for this generous contest, which brings important attention to the field of testing for AGI. >Happy to answer questions! 1. Can humans take the complete test suite? Has any human done so? Is it timed? How long does it take a human? What is the highest a human who sat down and took the ARC-AGI test scored? 2. How surprised would you be if a new model jumped to scoring 100% or nearly 100% on ARC-AGI (includin…
Humans can try the 800 tasks here. There is no time limit. I recommend not starting with the `expert` tasks, but instead go with the `entry` level puzzles. https://neoneye.github.io/arc/?dataset=ARC
If a model jumps to 100%, that may be a clever program or maybe the program has been trained on the 100 hidden tasks. Fchollet has 100 more hidden tasks, for verifying this.
Prediction markets on the outcome: https://manifold.markets/JacobPfau/will-the-arcagi-grand-pri...
https://manifold.markets/Tossup/will-the-arcagi-grand-prize-...
How can people believe that a censored politically correct process can get even close to something like AGI is baffling to me. Lysenkoism in computing.
This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and ha…
If a human eye works at say 10 fps, then 8 minutes with a cat is about 10k images :-D