The tests are only playable by people with normal color-vision.
Is there a "color-blind friendly" mode?
ARC Prize – a $1M+ competition towards open AGI progress
261–270 of 351 posts
Re: ARC Prize – a $1M+ competition towards open AGI progress
#262Earlier quoted context omitted.
> Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. I have kids so I'm presuming I'm allowed to have an opinion here. This is ignoring the fact that babies are not just learning labels, they're learning the whole of language, motion planning, sensory processing, etc. Once they have the basics down concept acquisition time shrinks rapidly and kids can easily learn their new fa…
> How many homework questions did your entire calc 1 class have? I'm guessing less than 100 and (hopefully) you successfully learned differential calculus. Not just that: people learn mathematics mainly by _thinking over and solving problems_, not by memorising solutions to problems. During my mathematics education I had to practice solving a lot of problems dissimilar what I had seen before. Even in the theory part,…
> Not just that: people learn mathematics mainly by _thinking over and solving problems_, not by memorising solutions to problems.
I think it's more accurate to say that they learn math by memorizing a sequence of steps that result in a correct solution, typically by following along with some examples. Hopefully they also remember why each step contributes to the answer as this aids recall and generalization.
The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly. This is just standard training. Understanding the motivation of each step helps with that memorization, and also allows you to apply that step in novel problems.
> The original article is spot on that there is no AGI pathway in the current research direction.
I think you're wrong. The research on grokking shows that LLMs transition from memorization to generalized circuits for problem solving if trained enough, and parametric memory generalizes their operation to many more tasks.
They have now been able to achieve near perfect accuracy on comparison tasks, where GPT-4 is barely in the double digit success rate.
Composition tasks are still challenging, but parametric memory is a big step in the right direction for that too. Accurate comparitive and compositional reasoning sound tantalizingly close to AGI.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#263Earlier quoted context omitted.
Did you try the puzzles?
No. What is the purpose of this competition? Unlikely that the reason for it is to pay out an enormous reward, right? Easy or not easy, the fortune is only rewarded to the system that solves the puzzles. The reward is too valuable to be given away easily. Ipso facto, solving the puzzles is deemed challenging by those who present the competition.
Are you writing this under every challenge with a monetary reward? The point of the challenge is that it is hard to do for an AI and easy for a human. Of course it is not easy to solve, that’s the point of the challenge. But the puzzle itself is not very hard.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#264Where did the money come from? How about put it toward alignment research instead of accelerating capabilities?
Exactly my thoughts...
Re: ARC Prize – a $1M+ competition towards open AGI progress
#265Earlier quoted context omitted.
Why not? If the only thing that can solve problem X is AGI (e.g. humans), and something else comes along that solves it, then rationally that should be evidence that the something else is AGI right? Unless you have strong prior beliefs (like "computers can't be AGI") or something else that's problem specific ("these problems can be solved by these techniques which don't count as AGI"). So I guess that's my real quest…
> If the only thing that can solve problem X is AGI (e.g. humans), and something else comes along that solves it, then rationally that should be evidence that the something else is AGI right? No. Because there might undiscovered ways to solve these problems that no one claims is AGI. The definition of AGI is notoriously fuzzy, but non-the-less if there was a 10 line python program (with no external dependencies or da…
I think I agree with you, but consider these two cases:
1. Only humans are known to have solved problem X, and we've spent no time looking for alternative solutions.
2. Only humans are known to have solved problem X, and we've spent hundreds of thousands of hours looking for alternative solutions and failed.
Now suppose something solves the problem. I feel like in case 2 we are justified in saying there's evidence that something is a human-like AGI. In case 1 we probably aren't justified in saying that.
To me this seems evident regardless of what the problem actually is! Because if it's hard enough that thousands of human hours cannot find a simple/algorithmic solution it's probably something like an "AGI-complete" problem?
Re: ARC Prize – a $1M+ competition towards open AGI progress
#266Earlier quoted context omitted.
> How many homework questions did your entire calc 1 class have? I'm guessing less than 100 and (hopefully) you successfully learned differential calculus. Not just that: people learn mathematics mainly by _thinking over and solving problems_, not by memorising solutions to problems. During my mathematics education I had to practice solving a lot of problems dissimilar what I had seen before. Even in the theory part,…
> Not just that: people learn mathematics mainly by _thinking over and solving problems_, not by memorising solutions to problems. I think it's more accurate to say that they learn math by memorizing a sequence of steps that result in a correct solution, typically by following along with some examples. Hopefully they also remember why each step contributes to the answer as this aids recall and generalization. The pra…
> The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly
Perhaps that is how you learned math, but it is nothing like how I learned math. Memorizing steps does not help, I sucked at it. What works for me us understanding the steps and why we used them. Once I understood the process and why it worked, I was able to reason my way through it.
> The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly.
Did you look at the types of problems presented by the ARC-AGO test? I don't see how memorization plays any role.
> They have now been able to achieve near perfect accuracy on comparison tasks, where GPT-4 is barely in the double digit success rate.
Then lets see how they do on the ARC test? While it is possible that generalized circuits can develop in Ls with enough training but I am pretty skeptical till we see results.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#267Earlier quoted context omitted.
> How many homework questions did your entire calc 1 class have? I'm guessing less than 100 and (hopefully) you successfully learned differential calculus. Not just that: people learn mathematics mainly by _thinking over and solving problems_, not by memorising solutions to problems. During my mathematics education I had to practice solving a lot of problems dissimilar what I had seen before. Even in the theory part,…
I think there is a component of memorizing solutions. For example, for mathematical proofs there is a set of standard "tricks" that you should have memorized.
Sure memory helps a lot, it allows you to concentrate your mental effort on the novel ot unique parts of the problem.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#268Earlier quoted context omitted.
> How many homework questions did your entire calc 1 class have? I'm guessing less than 100 and (hopefully) you successfully learned differential calculus. Not just that: people learn mathematics mainly by _thinking over and solving problems_, not by memorising solutions to problems. During my mathematics education I had to practice solving a lot of problems dissimilar what I had seen before. Even in the theory part,…
> Not just that: people learn mathematics mainly by _thinking over and solving problems_, not by memorising solutions to problems. I think it's more accurate to say that they learn math by memorizing a sequence of steps that result in a correct solution, typically by following along with some examples. Hopefully they also remember why each step contributes to the answer as this aids recall and generalization. The pra…
> The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly
Simply memorizing sequences of steps is not how mathematics learning works, otherwise we would not see so much variation in outcomes. Me and Terence Tao on the same exact math training data would not yield two mathematicians of similar skill.
While it's true that memorization of properties, structure, operations and what should be applied when and where is involved, there is a much deeper component of knowing how these all relate to each other. Grasping their fundamental meaning and structure, and some people seem to be wired to be better at thinking about and picking out these subtle mathematical relations using just the description or based off of only a few examples (or be able to at all, where everyone else struggles).
> I think you're wrong. The research on grokking shows that LLMs transition from memorization to generalized circuits
It's worth noting that for composition, key to abstract reasoning, LLMs failed to generalize to out of domain examples on simple synthetic data.
From: https://arxiv.org/abs/2405.15071
> The levels of generalization also vary across reasoning types: when faced with out-of-distribution examples, transformers fail to systematically generalize for composition but succeed for comparison.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#269Earlier quoted context omitted.
Why not? If the only thing that can solve problem X is AGI (e.g. humans), and something else comes along that solves it, then rationally that should be evidence that the something else is AGI right? Unless you have strong prior beliefs (like "computers can't be AGI") or something else that's problem specific ("these problems can be solved by these techniques which don't count as AGI"). So I guess that's my real quest…
Humans can do infinitely many things because we have general intelligence. Testing whether an AI can play chess or solve Chollet's ARC problems, or some other set of narrow skills, doesn't prove generality. If you want to test for generality, then you either have to: 1) Have a huge and very broad test suite, covering as many diverse human-level skills as possible. and/or, 2) Reductively understand what human intellig…
I suspect trying to reductively understand intelligence is a bit like trying to reductively understand biology - every level of abstraction is causally influenced by every other level of abstraction, so there just aren't simple primitives you can break everything down into.
Re: ARC Prize – a $1M+ competition towards open AGI progress
#270Earlier quoted context omitted.
> Now, is it 10k examples? No, but I think it was on the order of hundreds, if not thousands. I have kids so I'm presuming I'm allowed to have an opinion here. This is ignoring the fact that babies are not just learning labels, they're learning the whole of language, motion planning, sensory processing, etc. Once they have the basics down concept acquisition time shrinks rapidly and kids can easily learn their new fa…
> This is ignoring the fact that babies are not just learning labels, they're learning the whole of language, motion planning, sensory processing, etc. Yes. All that learning is feeding off one another. They're learning how reality works. Every bit of new information informs everything else. It's something that LLMs demonstrate too, so it shouldn't be a surprising observation. > Once they have the basics down concept…
> But that just means you have to compare them with ability of humans to one-shot tasks on the spot, "thinking on their feet", which for most tasks makes even adults look bad compared to GPT-4.
This simply is not true as stated in the article. ARC-AGI is a one-shot task test that humans reliably do much, much better on than any AI model.
> I don't believe someone could learn calc in 100 exercises or less.
I learned the basics of integration in a foreign language I barely understood by watching a couple of diagrams get drawn out and seeing far less than 100 examples or exercises.