I got the highest score on ARC-AGI again swapping Python for English • Hacker News Board

Re: I got the highest score on ARC-AGI again swapping Python for English

#11

Post by albertzeyer » Wed, Sep 17, 2025, 7:28 AM UTC

This sounds interesting.

I would really like to read a full research paper made out of this, which describes the method in more detail, gives some more examples, does more analysis on it, etc.

Btw, this uses LLMs on pure text-level? Why not images? Most of these patterns are easy to detect on image-level, but I assume when presented as text, it's much harder.

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible?

I think this argument is a bit flawed. Yes, you can define AGI as being better than (average) humans in every possible task. But isn't this very arbitrary? Isn't it more reasonable to expect that different intelligent systems (including animals, humans) can have different strengths, and it is unreasonable to expect that one system is really better in everything? Maybe it's more reasonable to define ASI that way, but even for ASI, if a system is already better in a majority of tasks (but not necessarily in every task), I think this should already count as ASI. Maybe really being better in every possible task is just not possible. You could design a task that is very specifically tailored for human intelligence.

Re: I got the highest score on ARC-AGI again swapping Python for English

#12

Post by harshitaneja » Wed, Sep 17, 2025, 7:31 AM UTC

sixo wrote: post #8

M4v3R wrote: post #6

Earlier quoted context omitted.

I wonder scaffolding synthesis is the way to go. Namely the LLM itself first reasons about the problem and creates scaffolding for a second agent that will do the actual solving. All inside a feedback loop to adjust the scaffolding based on results.

I toyed around with the idea of using an LLM to "compile" user instructions into a kind of AST of scaffolding, which can then be run by another LLM. It worked fairly wellbfor the kind of semi-structured tasks LLMs choke on like "for each of 100 things, do...", but I haven't taken it beyond a minimal impl.

I am working on something similar but with an AST for legal documents. So far, it seems promising but still rudimentary.

Re: I got the highest score on ARC-AGI again swapping Python for English

#13

Post by 0x20cowboy » Wed, Sep 17, 2025, 7:51 AM UTC

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible?

Because they are not.

Pattern matching questions on a contrived test is not the same thing as understanding or reasoning.

It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

Re: I got the highest score on ARC-AGI again swapping Python for English

#14

Post by wiseowise » Wed, Sep 17, 2025, 8:00 AM UTC

0x20cowboy wrote: post #13

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

[flagged]

Re: I got the highest score on ARC-AGI again swapping Python for English

#15

Post by justatdotin » Wed, Sep 17, 2025, 8:14 AM UTC

> LLMs have "dead reasoning zones" — areas in their weights where logic doesn't work. Humans have dead knowledge zones (things we don't know), but not dead reasoning zones.

blank stare

Re: I got the highest score on ARC-AGI again swapping Python for English

#16

Post by bubblyworld » Wed, Sep 17, 2025, 8:32 AM UTC

albertzeyer wrote: post #11

This sounds interesting. I would really like to read a full research paper made out of this, which describes the method in more detail, gives some more examples, does more analysis on it, etc. Btw, this uses LLMs on pure text-level? Why not images? Most of these patterns are easy to detect on image-level, but I assume when presented as text, it's much harder. > LLMs are PhD-level reasoners in math and science, yet th…

I suspect (to use the language of the author) current LLMs have a bit of a "reasoning dead zone" when it comes to images. In my limited experience they struggle with anything more complex than "transcribe the text" or similarly basic tasks. Like I tried to create an automated QA agent with Claude Sonnet 3.5 to catch regressions in my frontend, and it will look at an obviously broken frontend component (using puppeteer to drive and screenshot a headless browser) and confidently proclaim it's working correctly, often making up a supporting argument too. I've had much more success passing the code for the component and any console logs directly to the agent in text form.

My memory is a bit fuzzy, but I've seen another QA agent that takes a similar approach of structured text extraction rather than using images. So I suspect I'm not the only one finding image-based reasoning an issue. Could also be for cost reasons though, so take that with a pinch of salt.

Re: I got the highest score on ARC-AGI again swapping Python for English

#17

Post by bloqs » Wed, Sep 17, 2025, 8:39 AM UTC

wiseowise wrote: post #14

0x20cowboy wrote: post #13

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

[flagged]

please consider a less emotive, flaming/personal tone in the future, hacker news is much more readable without it!

I would broadly agree that it's a bit far, but the OPs point does have some validity, its often the same formulaic methodology

Re: I got the highest score on ARC-AGI again swapping Python for English

#18

Post by mjburgess » Wed, Sep 17, 2025, 9:13 AM UTC

justatdotin wrote: post #15

> LLMs have "dead reasoning zones" — areas in their weights where logic doesn't work. Humans have dead knowledge zones (things we don't know), but not dead reasoning zones. blank stare

We have dead-zones in adductive reasoning, not in induction or deduction. Almost all failures of reasoning in people are in abducing what model describes the situation at hand.

eg., we can apply the rule, "-A cannot follow from A", etc. regardless of the A

eg., we always know that if the number of apples is 2, then it cannot be any of "all numbers without 2" -- which quantifies over all numbers

You will not find a "gap" for a given number, whereas with LLMs, gaps of this kind are common

Re: I got the highest score on ARC-AGI again swapping Python for English

#19

Post by didroe » Wed, Sep 17, 2025, 9:20 AM UTC

>With RL, models no longer just learn what sounds correct based on patterns they've seen. They learn what words to output to be correct. RL is the process of forcing the pre-trained weights to be logically consistent.

How does Reinforcement Learning force the weights to be logically consistent? Isn't it just about training using a coarser/more-fuzzy granularity of fitness?

More generally, is it really solving the task if it's given a large number of attempts and an oracle to say whether it's correct? Humans can answer the questions in one shot and self-check the answer, whereas this is like trial and error with an external expert who tells you to try again.

Re: I got the highest score on ARC-AGI again swapping Python for English

#20

Post by sunrunner » Wed, Sep 17, 2025, 9:26 AM UTC

modeless wrote: post #5

I've been testing LLMs on Sokoban-like puzzles (in the style of ARC-AGI-3) and they are completely awful at them. It really highlights how poor their memory is. They can't remember abstract concepts or rules between steps, even if they discover them themselves. They can only be presented with lossy text descriptions of such things which they have to re-read and re-interpret at every step. LLMs are completely helpless…

I'm not sure how similar this is but I tried the same quite a while back with a simple 5x5 nonogram (Picross) and had similar difficulties.

I found not only incorrect 'reasoning' but also even after being explicit about why a certain deduction was not correct the same incorrect deduction would then appear later, and this happened over and over.

Also, there's already a complete database of valid answers at [1], so I'm not sure why the correct answer couldn't just come from that, and the 'reasoning' can be 'We solved this here, look...' ;)

[1] The wonderful https://pixelogic.app/every-5x5-nonogram