Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I got the highest score on ARC-AGI again swapping Python for English

jeremyberman.substack.com

Post reply on HN
Print view View original

131–136 of 136 posts

Re: I got the highest score on ARC-AGI again swapping Python for English

#131

Post by Grimblewald »

0x20cowboy wrote: post #13

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

I fully agree, and any doubters, I can assure you that an LLM is bested by most undergraduates when it comes to reasoning. LLM's will universally get smoked by any PhD. They have great big wealth of knowledge to draw on but critical thinking is sorely lacking.

LLM's strength is being an interactive encyclopedia, not a decision making thing.

Re: I got the highest score on ARC-AGI again swapping Python for English

#132

Post by Grimblewald »

gwd wrote: post #23
0x20cowboy wrote: post #13

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

> Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. Do submarines swim? I don't really care if it gets me where I want to go. The fact is that just two days ago, I asked Claude to look at some reasonably complicated concurrent code to which I had added a new feature, and asked it to list what tests needed to be added; and then when I asked GPT-5 to add them, it one-sh…

> just read the description

Seems fine enough to me. Wanna really challenge an LLM? get it to make an image stitching algorithm that isn't shit. Implement the results from brown et al https://link.springer.com/article/10.1007/s11263-006-0002-3 and I'll be impressed.

This is a paper from 2007 and there are plenty of packages available to help make it all happen through some API calls and a bit of cleverness on the coders part, and so far not a single LLM has gotten close to an acceptable implementation. Not a single one.

Now, why is it so hard? Because there's not public code for good quality high performance image stitching on the level of the image composite editor microsoft research once hosted. There's nothing for the LLM's to draw on and they fundamentally lack reasoning / planning other than something that superficially resembles it, but it falls apart for out of domain things where humans still do fine even if new to the task.

Re: I got the highest score on ARC-AGI again swapping Python for English

#133

Post by naasking »

HarHarVeryFunny wrote: post #126
naasking wrote: post #125

Earlier quoted context omitted.

> Powerful pattern matching is still just pattern matching. Higher order pattern matching is Turing complete. Transformers are Turing complete. Memory augmented LLMs are Turing complete. Neural networks can learn to reproduce any function. These have all been proven. So if computers can be intelligent and can solve novel problems in principle, then LLMs can too if given the right training. If you don't think computer…

Being Turing machine complete means that the system in question can emulate a Turing machine, which you could then program to do anything since it's a universal computer. So sure, if you know how to code up an AGI to run on a Turing machine you would be good to go on any Turing machine! I'm not sure why you want to run a Turing machine emulator on an LLM, when you could just write a massively faster one to run on the…

The point has nothing to do with speed, but with expressive power / what is achievable and learnable, in principle. Again, if you accept that a computer can in principle run a program that qualifies as AGI, then all I'm saying is that an LLM with memory augmentation can in principle be trained to do this as well because their computation power is formally equivalent.

And coincidentally, a new paper being discussed on HN is a good example addressing your concern about existing models learning and developing novel things. Here's a GPT model that learned physics just by training on a data:

https://arxiv.org/abs/2509.13805

Re: I got the highest score on ARC-AGI again swapping Python for English

#134

Post by HarHarVeryFunny »

naasking wrote: post #133
HarHarVeryFunny wrote: post #126

Earlier quoted context omitted.

Being Turing machine complete means that the system in question can emulate a Turing machine, which you could then program to do anything since it's a universal computer. So sure, if you know how to code up an AGI to run on a Turing machine you would be good to go on any Turing machine! I'm not sure why you want to run a Turing machine emulator on an LLM, when you could just write a massively faster one to run on the…

The point has nothing to do with speed, but with expressive power / what is achievable and learnable, in principle. Again, if you accept that a computer can in principle run a program that qualifies as AGI, then all I'm saying is that an LLM with memory augmentation can in principle be trained to do this as well because their computation power is formally equivalent. And coincidentally, a new paper being discussed on…

You seem to want to say that because an LLM is Turing complete (a doubtful claim) it should be able to implement AGI, which would be a logical conclusion, but yet totally irrelevant.

If the only thing missing to implement AGI was a Turing machine to run it on, then we'd already have AGI running on Conway's game of Life, or perhaps on a Google supercomputer.

> Here's a GPT model that learned physics just by training on a data

It didn't learn at run-time. It was PRE-trained, using SGD on the entire training set, the way that GPT's (Generative PRE-trained Transformers) always are.

In order to learn at run-time, or better yet get rid of the distinction between pre-training and run-time, requires someone to invent (or copy from nature) a new incremental learning algorithm that:

a) Doesn't require retraining on everything it was ever previously trained on, and

b) Doesn't cause it to forget, or inappropriately change, things it had previously learnt

These are easier said than done, which is why we're a decade or so into the "deep learning" revolution, and nothing much has changed other than fine-tuning which is still a bulk data technique.

Re: I got the highest score on ARC-AGI again swapping Python for English

#135

Post by washadjeffmad »

gwd wrote: post #23
0x20cowboy wrote: post #13

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

> Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. Do submarines swim? I don't really care if it gets me where I want to go. The fact is that just two days ago, I asked Claude to look at some reasonably complicated concurrent code to which I had added a new feature, and asked it to list what tests needed to be added; and then when I asked GPT-5 to add them, it one-sh…

I tend to skip these threads because there's a gulf in understanding between those who have accepted the marketing and those who are hoping to profit from them.

"PhD level reasoning" means they have asked PhDs questions and finetuned on their responses. It does not mean that every response is "PhD level", that the models only provide responses that PhDs have validated, or that every response is correct. It is "Lysol kills 99.9% of germs" logic.

The domains I work in and contribute to largely view being paid to write out their thought processes when answering questions models answer incorrectly as a novelty, like grading homework with some minor bragging rights.

Re: I got the highest score on ARC-AGI again swapping Python for English

#136

Post by kandel »

rel_ic wrote: post #51
mjburgess wrote: post #34

Earlier quoted context omitted.

That's an abductive failure to recognise that something is A, and something else is not-A I dont see cases where people recognise the contradiction and then perform it.

People who know alcohol is bad for them and don't want to keep being drunks but keep drinking, people who believe phones are bad for their kids but still buy them, people who understand AI will significantly degrade the environment if it becomes ubiquitous but still work to help it become ubiquitous... Mathematicians who publish proofs that are later proven inconsistent! I suspect we have fundamentally different view…

Well most of your examples are about failure to act based on reasoning rather than failure to reason, except the mathematics one which is unfair as research mathematics is a very hard task - either subtle errors, reasoning in new fields, or extremely long chains of reasoning.
Post reply on HN