Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I got the highest score on ARC-AGI again swapping Python for English

jeremyberman.substack.com

Post reply on HN
Print view View original

31–40 of 136 posts

Re: I got the highest score on ARC-AGI again swapping Python for English

#31

Post by Akronymus »

gwd wrote: post #23
0x20cowboy wrote: post #13

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

> Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. Do submarines swim? I don't really care if it gets me where I want to go. The fact is that just two days ago, I asked Claude to look at some reasonably complicated concurrent code to which I had added a new feature, and asked it to list what tests needed to be added; and then when I asked GPT-5 to add them, it one-sh…

> I just don't see how you can argue with a straight face that this is "pattern matching". If that's pattern matching, then pattern matching is not an insult.

IMO its still "just" a, very good, autocomplete. No actual reasoning, but lots of statistics on what is the next token to spit out.

Re: I got the highest score on ARC-AGI again swapping Python for English

#32

Post by Garlef »

That's a super neat approach.

But the core issue seems to be: How do you come up with the fitness function that drives the evolutionary process without human intervention in the first place?

(I've tried something similar with a coding agent where I let the agent modify parts of its system prompt... But it got stuck very fast since there was no clear fitness function)

Re: I got the highest score on ARC-AGI again swapping Python for English

#33

Post by Akronymus »

sunrunner wrote: post #20
modeless wrote: post #5

I've been testing LLMs on Sokoban-like puzzles (in the style of ARC-AGI-3) and they are completely awful at them. It really highlights how poor their memory is. They can't remember abstract concepts or rules between steps, even if they discover them themselves. They can only be presented with lossy text descriptions of such things which they have to re-read and re-interpret at every step. LLMs are completely helpless…

I'm not sure how similar this is but I tried the same quite a while back with a simple 5x5 nonogram (Picross) and had similar difficulties. I found not only incorrect 'reasoning' but also even after being explicit about why a certain deduction was not correct the same incorrect deduction would then appear later, and this happened over and over. Also, there's already a complete database of valid answers at [1], so I'm…

> I found not only incorrect 'reasoning' but also even after being explicit about why a certain deduction was not correct the same incorrect deduction would then appear later, and this happened over and over.

Because its in the context window and a lot of training material refers to earlier stuff for later stuff it is trained to bring up that stuff again and again. Even if it is in the window as a negative.

Re: I got the highest score on ARC-AGI again swapping Python for English

#34

Post by mjburgess »

rel_ic wrote: post #30
mjburgess wrote: post #18

Earlier quoted context omitted.

We have dead-zones in adductive reasoning, not in induction or deduction. Almost all failures of reasoning in people are in abducing what model describes the situation at hand. eg., we can apply the rule, "-A cannot follow from A", etc. regardless of the A eg., we always know that if the number of apples is 2, then it cannot be any of "all numbers without 2" -- which quantifies over all numbers You will not find a "g…

> we can apply the rule, "-A cannot follow from A", etc. regardless of the A You can't think of any domains where we are unable to apply this rule? I feel like I'm surrounded by people claiming "A, therefore -A!!" And if I'm one of them, and this were a reasoning dead-zone for me, I wouldn't be able to tell!

That's an abductive failure to recognise that something is A, and something else is not-A

I dont see cases where people recognise the contradiction and then perform it.

Re: I got the highest score on ARC-AGI again swapping Python for English

#35

Post by NoahZuniga »

Akronymus wrote: post #31
gwd wrote: post #23

Earlier quoted context omitted.

> Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. Do submarines swim? I don't really care if it gets me where I want to go. The fact is that just two days ago, I asked Claude to look at some reasonably complicated concurrent code to which I had added a new feature, and asked it to list what tests needed to be added; and then when I asked GPT-5 to add them, it one-sh…

> I just don't see how you can argue with a straight face that this is "pattern matching". If that's pattern matching, then pattern matching is not an insult. IMO its still "just" a, very good, autocomplete. No actual reasoning, but lots of statistics on what is the next token to spit out.

> Do submarines swim?

That's the main point of the parent comment. Arguing about the definition of "reasoning" or "pattern matching" is just a waste of time. What really matters is if it produces helpful output. Arguing about that is way better!

Instead of saying: "It's just pattern matching -> It won't improve the world", make an argument like: "AI's seem to have trouble specializing like humans -> adopting AI will increase error rates in business processes -> due to the amount of possible edge cases, most people will get into an edge case with no hope of escaping it -> many people's lives will get worse".

The first example relies on us agreeing on the definition of pattern matching, and then taking a conclusion based on how those words feel. This has no hope of convincing me if I don't like your definition! The second one is an argument that could potentially convince me, even if I'm an AI optimist. It is also just by itself an interesting line of reasoning.

Re: I got the highest score on ARC-AGI again swapping Python for English

#36

Post by ACCount37 »

bubblyworld wrote: post #16
albertzeyer wrote: post #11

This sounds interesting. I would really like to read a full research paper made out of this, which describes the method in more detail, gives some more examples, does more analysis on it, etc. Btw, this uses LLMs on pure text-level? Why not images? Most of these patterns are easy to detect on image-level, but I assume when presented as text, it's much harder. > LLMs are PhD-level reasoners in math and science, yet th…

I suspect (to use the language of the author) current LLMs have a bit of a "reasoning dead zone" when it comes to images. In my limited experience they struggle with anything more complex than "transcribe the text" or similarly basic tasks. Like I tried to create an automated QA agent with Claude Sonnet 3.5 to catch regressions in my frontend, and it will look at an obviously broken frontend component (using puppetee…

LLM image frontends suck, and a lot of them suck big time.

The naive approach of "use a pretrained encoder to massage the input pixels into a bag of soft tokens and paste those tokens into the context window" is good enough to get you a third of the way to humanlike vision performance - but struggles to go much further.

Claude's current vision implementation is also notoriously awful. Like, "a goddamn 4B Gemma 3 beats it" level of awful. For a lot of vision-heavy tasks, you'd be better off using literally anything else.

Re: I got the highest score on ARC-AGI again swapping Python for English

#37

Post by ozgung »

Akronymus wrote: post #31
gwd wrote: post #23

Earlier quoted context omitted.

> Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. Do submarines swim? I don't really care if it gets me where I want to go. The fact is that just two days ago, I asked Claude to look at some reasonably complicated concurrent code to which I had added a new feature, and asked it to list what tests needed to be added; and then when I asked GPT-5 to add them, it one-sh…

> I just don't see how you can argue with a straight face that this is "pattern matching". If that's pattern matching, then pattern matching is not an insult. IMO its still "just" a, very good, autocomplete. No actual reasoning, but lots of statistics on what is the next token to spit out.

No it's not "just a very good autocomplete". I don't know why people repeat this thing (it's wrong) but I find it an extremely counterproductive position. Some people just love to dismiss the capabilities of AI with a very shallow understanding of how it works. Why?

It generates words one by one, like we all do. This doesn't mean it does just that and nothing else. It's the mechanics of how they are trained and how they do inference. And most importantly how they communicate with us. It doesn't define what they are or their limits. This is reductionism. Ignoring the mathematical complexity of a giant neural network.

Re: I got the highest score on ARC-AGI again swapping Python for English

#38

Post by virgilp »

mjburgess wrote: post #34
rel_ic wrote: post #30

Earlier quoted context omitted.

> we can apply the rule, "-A cannot follow from A", etc. regardless of the A You can't think of any domains where we are unable to apply this rule? I feel like I'm surrounded by people claiming "A, therefore -A!!" And if I'm one of them, and this were a reasoning dead-zone for me, I wouldn't be able to tell!

That's an abductive failure to recognise that something is A, and something else is not-A I dont see cases where people recognise the contradiction and then perform it.

How can you know? One could argue that the entire phenomenon of cognitive dissonance is "people (internally) recognize the contradiction and then perform it"

Re: I got the highest score on ARC-AGI again swapping Python for English

#39

Post by FergusArgyll »

The biggest issue I have with ARC-AGI is it's a visual problem. LLMs (even the newfangled multi-modal ones) are still far worse at vision than at purely text based problems. I don't think it's possible to build a test of purely text-based questions that would be easy for humans and hard for SOTA models. Yes, there's a few gotchas you can throw at them but not 500.

Re: I got the highest score on ARC-AGI again swapping Python for English

#40

Post by [deleted] »

ozgung wrote: post #37
Akronymus wrote: post #31

Earlier quoted context omitted.

> I just don't see how you can argue with a straight face that this is "pattern matching". If that's pattern matching, then pattern matching is not an insult. IMO its still "just" a, very good, autocomplete. No actual reasoning, but lots of statistics on what is the next token to spit out.

No it's not "just a very good autocomplete". I don't know why people repeat this thing (it's wrong) but I find it an extremely counterproductive position. Some people just love to dismiss the capabilities of AI with a very shallow understanding of how it works. Why? It generates words one by one, like we all do. This doesn't mean it does just that and nothing else. It's the mechanics of how they are trained and how t…

[deleted]
Post reply on HN