Live data from Hacker News

I got the highest score on ARC-AGI again swapping Python for English

jeremyberman.substack.com

121–130 of 136 posts

Re: I got the highest score on ARC-AGI again swapping Python for English

#121

Earlier quoted context omitted.

The primary source is: measured LLM performance on once-human-exclusive tasks - such as high end natural language processing or commonsense reasoning. Those things were once thought to require a human mind - clearly, not anymore. Human commonsense knowledge can be both captured and applied by a learning algorithm trained on nothing but a boatload of text. But another important source is: loads and loads of mech inter…

I haven't seen LLMs perform common sense reasoning. Feel free to share some links. Your post reads like anthropomorphized nonsense.

One of the most astonishing things about LLMs is that they actually seem to have achieved general common-sense reasoning to a signficant extent. Example from the thread about somebody ordering 18000 waters at a drive-through: https://news.ycombinator.com/item?id=45067653

TL;DR: Even without being explicitly prompted to, a pretty weak LLM "realized" that a thousand glasses of water was an unreasonable order. I'd say that's good enough to call "common sense".

You can try it out yourself! Just pick any AI chatbot, make up situations with varying levels of absurdity, maybe in a roleplay setting (e.g. "You are a fast food restaurant cashier. I am a customer. My order is..."), and test how it responds.

Re: I got the highest score on ARC-AGI again swapping Python for English

#122

Earlier quoted context omitted.

Neuroscience is the field that would be closest to this. But even they are empty handed with evidence and heavy with hypotheses.

No, psychology is right. Psychology studies what the properties of thought are. Neuroscience studies the specific biochemical mechanisms of the brain. Psychology is the study of what mental reasoning IS, while neuroscience is the study of HOW neurons in our brain implement it. If you are asking “ok, but what is reasoning, really? What definition of reasoning would enable us to recognize whether it is going on in this…

Psychology is stuck in pre-Galilean era. Even if it studies "properties of thought", as you put it, it does so without formal basis, let alone understanding from first principles. As Chomsky said, about psychology and the like, "You want to move from behavioral science to authentic science." [1]

[1] Chomsky & Krauss (2015) An Origins Project Dialogue at https://youtu.be/Ml1G919Bts0

Re: I got the highest score on ARC-AGI again swapping Python for English

#123
post #21

isn't the author actually overfitting a solution ? He'll sure beat ARC AGI, but that will be all.

Seeing how ARC-AGI is pretty much the only non-embodied short-duration type of challenge where humans are still an order of magnitude better than AIs, beating it would possibly bring us a lot closer to actual AGI.

Re: I got the highest score on ARC-AGI again swapping Python for English

#124
post #37

Earlier quoted context omitted.

> I just don't see how you can argue with a straight face that this is "pattern matching". If that's pattern matching, then pattern matching is not an insult. IMO its still "just" a, very good, autocomplete. No actual reasoning, but lots of statistics on what is the next token to spit out.

No it's not "just a very good autocomplete". I don't know why people repeat this thing (it's wrong) but I find it an extremely counterproductive position. Some people just love to dismiss the capabilities of AI with a very shallow understanding of how it works. Why? It generates words one by one, like we all do. This doesn't mean it does just that and nothing else. It's the mechanics of how they are trained and how t…

I don't know why people repeat this thing (it's wrong)

Because they simply don't care if they're wrong. At this point, given what we've seen, that seems like the only explanation left.

You don't need to be a fanatical AGI evangelist, but when an "autocomplete" starts winning international math competitions, you need to start calling it something else.

Re: I got the highest score on ARC-AGI again swapping Python for English

#125

Earlier quoted context omitted.

> but rather the ability to reason in the general case, which requires the ability to LEARN to solve novel problems, which is what is missing from LLMs. I don't think it's missing, zero shot prompting is quite successful in many cases. Maybe you find the extent that LLMs can do this to be too limited, but I'm not sure that means they don't reason at all. > A system that has a fixed set of (reasoning/prediction) rules…

Powerful pattern matching is still just pattern matching. How is an LLM going to solve a novel problem with just pattern matching? Novel means it has never seen it before, maybe doesn't even have the knowledge needed to solve it, so it's not going to be matching any pattern, and even if it did, that would not help if it required a solution different to whatever the pattern match had come from. Human level reasoning i…

> Powerful pattern matching is still just pattern matching.

Higher order pattern matching is Turing complete. Transformers are Turing complete. Memory augmented LLMs are Turing complete. Neural networks can learn to reproduce any function. These have all been proven.

So if computers can be intelligent and can solve novel problems in principle, then LLMs can too if given the right training. If you don't think computers can be intelligent, you have a much higher burden to meet.

> Human level reasoning includes ability to learn, so that people can solve novel problems, overcome failures by trial and error, exploration, etc.

You keep bringing this up as if it's lacking, but basically all existing LLM interfaces provide facilities for memory to store state. Storing progress just isn't an issue if the LLM has the right training. HN has some recent articles about Claude code just being given the task to port some GitHub repos to other programming languages, and they woke up the next morning and it did it autonomously, using issue tracking, progress reports, PRs the hole nine yards. This is frankly not the hard part IMO.

Re: I got the highest score on ARC-AGI again swapping Python for English

#126

Earlier quoted context omitted.

Powerful pattern matching is still just pattern matching. How is an LLM going to solve a novel problem with just pattern matching? Novel means it has never seen it before, maybe doesn't even have the knowledge needed to solve it, so it's not going to be matching any pattern, and even if it did, that would not help if it required a solution different to whatever the pattern match had come from. Human level reasoning i…

> Powerful pattern matching is still just pattern matching. Higher order pattern matching is Turing complete. Transformers are Turing complete. Memory augmented LLMs are Turing complete. Neural networks can learn to reproduce any function. These have all been proven. So if computers can be intelligent and can solve novel problems in principle, then LLMs can too if given the right training. If you don't think computer…

Being Turing machine complete means that the system in question can emulate a Turing machine, which you could then program to do anything since it's a universal computer. So sure, if you know how to code up an AGI to run on a Turing machine you would be good to go on any Turing machine!

I'm not sure why you want to run a Turing machine emulator on an LLM, when you could just write a massively faster one to run on the computer your LLM is running on, cutting out the middle man, but whatever floats your boat I suppose.

Heck, if you really like emulation and super slow speed then how about implementing Conway's game of Life to run on your LLM Turing machine emulator, and since Life is also Turing complete you could run another Turing machine emulator on that (it's been done), and finally run your AGI on top of that! Woo hoo!

I do think you'll have a challenge prompting your LLM to emulate a Turing machine (they are really not very good at that sort of thing), especially since the prompt/context will also have to do double duty as the Turing machines (infinite length) tape, but no doubt you'll figure it out.

Keep us posted.

I'll be excited to see your AGI program when you write that bit.

Re: I got the highest score on ARC-AGI again swapping Python for English

#127

Can someone explain to me why a new LLMs ability to solve highly publicized puzzles is not "just" (sorry) it having access to the blog posts talking about those puzzles? It's fine, that's what I would do to solve them, but it doesn't obviously and immediately make me confident in new reasoning capability w that suspicion floating around.

Because people already tried to get LLMs to solve ARC-AGI puzzles by training on millions of similar puzzles, and it doesn’t work.

Some problems fundamentally require many serial steps to solve. Reasoning LLMs can work through those steps, base LLMs can’t.

Re: I got the highest score on ARC-AGI again swapping Python for English

#128
post #109
post #68

Earlier quoted context omitted.

It's trivial to demonstrate that LLMs are pattern matching rather than reasoning. A good way is to provide modified riddles-that-aren't. As an example: > Prompt: A man working at some white collar job gets an interview scheduled with an MBA candidate. The man says "I can't interview this candidate, he's my son." How is this possible? > ChatGPT: Because the interviewer is the candidate’s mother. (The riddle plays on t…

We kinda move from the situation “LLM can only do what it seen before” to “LLM can do something by composing several things it has seen before”. We didn’t get to the situation “LLM can do things it has not seen before”. The practicality of the situation is that a lot of problems fall into the second bucket. We all like to think we deal with novel problems, but most of what we can think of was already considered by an…

I routinely use LLMs to do things that have never been done before. It requires carefully structured prompting and context management, but it is quite doable.

Re: I got the highest score on ARC-AGI again swapping Python for English

#129

To me the reason ARC-AGI puzzles are difficult for LLMs and possible for humans is that they are expressed in a format for which humans have powerful preprocessing capabilities. Imagine the puzzle layouts were expressed in JSON instead of as a pattern of visual blocks. How many humans could solve them in that case?

[dead]

Re: I got the highest score on ARC-AGI again swapping Python for English

#130
post #21

isn't the author actually overfitting a solution ? He'll sure beat ARC AGI, but that will be all.

Seeing how ARC-AGI is pretty much the only non-embodied short-duration type of challenge where humans are still an order of magnitude better than AIs, beating it would possibly bring us a lot closer to actual AGI.

https://arxiv.org/html/2505.07859v1
Post reply on HN