isn't the author actually overfitting a solution ? He'll sure beat ARC AGI, but that will be all.
I got the highest score on ARC-AGI again swapping Python for English
21–30 of 136 posts
Re: I got the highest score on ARC-AGI again swapping Python for English
#22> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.
"Not understanding or reasoning" is anthropocentric cope. There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs.
One notable difference, however, is that LLMs disproportionately suck at spatial reasoning. Which shouldn't be surprising, considering that their training datasets are almost entirely text. The ultimate wordcel makes for a poor shape rotator.
All ARC-AGI tasks are "spatial reasoning" tasks. They aren't in any way special. They just force LLMs to perform in an area they're spectacularly weak at. And LLMs aren't good enough yet to be able to brute force through this innate deficiency with raw intelligence.
Re: I got the highest score on ARC-AGI again swapping Python for English
#23> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.
> Pattern matching questions on a contrived test is not the same thing as understanding or reasoning.
Do submarines swim? I don't really care if it gets me where I want to go. The fact is that just two days ago, I asked Claude to look at some reasonably complicated concurrent code to which I had added a new feature, and asked it to list what tests needed to be added; and then when I asked GPT-5 to add them, it one-shot nailed the implementations. I've written a gist of it here:
https://gitlab.com/-/snippets/4889253
Seriously just even read the description of the test it's trying to write.
In order to one-shot that code, it had to understand:
- How the cache was supposed to work
- How conceptually to set up the scenario described
- How to assemble golang's concurrency primitives (channels, goroutines, and waitgroups), in the correct order, to achieve the goal.
Did it have a library of concurrency testing patterns in its head? Probably -- so do I. Had it ever seen my exact package before in its training? Never.
I just don't see how you can argue with a straight face that this is "pattern matching". If that's pattern matching, then pattern matching is not an insult.
If anything, the examples in this article are the opposite. Take the second example, which is basically 'assemble these assorted pieces into a rectangle'. Nearly every adult has assembled a minimum of dozens of things in their lives; many have assembled thousands of things. So it's humans in this case who are simply "pattern matching questions on a contrived test", and the LLMs, which almost certainly didn't have a lot of "assemble these items" in their training data, that are reasoning out what's going on from first principles.
Re: I got the highest score on ARC-AGI again swapping Python for English
#24This sounds like it is just slightly smarter than brute forcing your way to a solution.
Oh well, more support for my prediction: nobody will win a Nobel prize for reaching AGI.
Re: I got the highest score on ARC-AGI again swapping Python for English
#25Congrats, you made LLMs perform slightly better at a contrived puzzle. This finally proves that we've cracked intelligence and are well on our way towards AGI.
Re: I got the highest score on ARC-AGI again swapping Python for English
#26> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.
"Not understanding or reasoning" is anthropocentric cope. There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs. One notable difference, however, is that LLMs disproportionately suck at spatial reasoning . Which shouldn't be surprising, considering that their training datasets are almost entirely text. The ultimate wordcel makes for a…
> There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs.
Source?
Re: I got the highest score on ARC-AGI again swapping Python for English
#27Earlier quoted context omitted.
"Not understanding or reasoning" is anthropocentric cope. There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs. One notable difference, however, is that LLMs disproportionately suck at spatial reasoning . Which shouldn't be surprising, considering that their training datasets are almost entirely text. The ultimate wordcel makes for a…
> There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs. Source?
...literally benchmarks the post is all about?
practical difference is about results - and results are here
Re: I got the highest score on ARC-AGI again swapping Python for English
#28isn't the author actually overfitting a solution ? He'll sure beat ARC AGI, but that will be all.
I don't think so. The author isn't training an LLM, but rather using an LLM to solve a specific problem. This method could also be applied to solve other problems.
Re: I got the highest score on ARC-AGI again swapping Python for English
#29Earlier quoted context omitted.
"Not understanding or reasoning" is anthropocentric cope. There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs. One notable difference, however, is that LLMs disproportionately suck at spatial reasoning . Which shouldn't be surprising, considering that their training datasets are almost entirely text. The ultimate wordcel makes for a…
> There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs. Source?
The primary source is: measured LLM performance on once-human-exclusive tasks - such as high end natural language processing or commonsense reasoning.
Those things were once thought to require a human mind - clearly, not anymore. Human commonsense knowledge can be both captured and applied by a learning algorithm trained on nothing but a boatload of text.
But another important source is: loads and loads of mech interpret research that tried to actually pry the black box open and see what happens on the inside.
This found some amusing artifacts - such as latent world models that can be extracted from the hidden state, or neural circuits corresponding to high level abstracts being chained together to obtain the final outputs. Very similar to human "abstract thinking" in function - despite being implemented on a substrate of floating point math and not wet meat.
Re: I got the highest score on ARC-AGI again swapping Python for English
#30> LLMs have "dead reasoning zones" — areas in their weights where logic doesn't work. Humans have dead knowledge zones (things we don't know), but not dead reasoning zones. blank stare
We have dead-zones in adductive reasoning, not in induction or deduction. Almost all failures of reasoning in people are in abducing what model describes the situation at hand. eg., we can apply the rule, "-A cannot follow from A", etc. regardless of the A eg., we always know that if the number of apples is 2, then it cannot be any of "all numbers without 2" -- which quantifies over all numbers You will not find a "g…
> we can apply the rule, "-A cannot follow from A", etc. regardless of the A
You can't think of any domains where we are unable to apply this rule? I feel like I'm surrounded by people claiming "A, therefore -A!!"
And if I'm one of them, and this were a reasoning dead-zone for me, I wouldn't be able to tell!