Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I got the highest score on ARC-AGI again swapping Python for English

jeremyberman.substack.com

Post reply on HN
Print view View original

71–80 of 136 posts

Re: I got the highest score on ARC-AGI again swapping Python for English

#71

Post by bubblyworld »

ACCount37 wrote: post #36
bubblyworld wrote: post #16

Earlier quoted context omitted.

I suspect (to use the language of the author) current LLMs have a bit of a "reasoning dead zone" when it comes to images. In my limited experience they struggle with anything more complex than "transcribe the text" or similarly basic tasks. Like I tried to create an automated QA agent with Claude Sonnet 3.5 to catch regressions in my frontend, and it will look at an obviously broken frontend component (using puppetee…

LLM image frontends suck, and a lot of them suck big time. The naive approach of "use a pretrained encoder to massage the input pixels into a bag of soft tokens and paste those tokens into the context window" is good enough to get you a third of the way to humanlike vision performance - but struggles to go much further. Claude's current vision implementation is also notoriously awful. Like, "a goddamn 4B Gemma 3 beat…

Wild, I found it hard to believe that a 4b model could beat sonnet-3.5 at anything, but at least on the vision arena (https://lmarena.ai/leaderboard/vision) it seems like sonnet-3.5 is at the same ELO as a 27b gemma (~1150), so it's plausible. I guess that just says more about how bad vision LLMs are right now that anything else.

Re: I got the highest score on ARC-AGI again swapping Python for English

#72

Post by naasking »

mjr00 wrote: post #68
naasking wrote: post #60

Earlier quoted context omitted.

> are they really learning to reason, or are they just learning to pattern match to steer generation in the direction of problem-specific reasoning steps that they had been trained on? Are you sure there's a real difference? Do you have a definition of "reasoning" that excludes this?

It's trivial to demonstrate that LLMs are pattern matching rather than reasoning. A good way is to provide modified riddles-that-aren't. As an example: > Prompt: A man working at some white collar job gets an interview scheduled with an MBA candidate. The man says "I can't interview this candidate, he's my son." How is this possible? > ChatGPT: Because the interviewer is the candidate’s mother. (The riddle plays on t…

> It's trivial to demonstrate that LLMs are pattern matching rather than reasoning.

Again, this is just asserting the premise that reasoning cannot include pattern matching, but this has never been justified. What is your definition for "reasoning"?

> This is clearly pattern matching and overfitting to the "doctor riddle" and a good demonstration of how there's no actual reasoning going on.

Not really, no. "Bad reasoning" does not entail "no reasoning". Your conclusion is simply too strong for the evidence available, which is why I'm asking for a rigourous definition of reasoning that doesn't leave room for disagreement about whether pattern matching counts.

Re: I got the highest score on ARC-AGI again swapping Python for English

#73

Post by HarHarVeryFunny »

gwd wrote: post #67
HarHarVeryFunny wrote: post #54

Earlier quoted context omitted.

> Do submarines swim? It doesn't matter HOW LLMs "swim" as long as they can, but the point being raised is whether they actually can. It's as if LLMs can swim in the ocean, in rough surf, but fail to swim in rivers or swimming pools, because they don't have a generalized ability to swim - they've just been RL-trained on the solution steps to swimming in surf, but since those exact conditions don't exist in a river (w…

> I think the answer is pretty obvious given that LLM's can't learn at runtime - can't try out some reasoning generalization they may have arrived at, find that it doesn't work in a specific case, then explore the problem and figure it out for next time. This is just a problem of memory. Supposing that an LLM did generate a genuinely novel insight, it could in theory they could write a note for itself so that next ti…

> This is just a problem of memory

How does memory (maybe later incorporated via fine tuning) help if you can't figure out how to do something in the first place ?

That would be a way to incorporate new declarative data at "runtime" - feedback to the AI intern as to what it is doing wrong. However, in order to do something effectively by yourself generally requires more than just new knowledge - it requires personal practice/experimentation etc, since you need to learn how to act based on the contents of your own mind, not that of the instructor.

Even when you've had enough practice to become proficient at a taught skill, you may not be able to verbalize exactly what you are doing (which is part of the teacher-student gap), so attempting to describe then capture that as textual/context "sensory input" is not always going to work.

Re: I got the highest score on ARC-AGI again swapping Python for English

#74

Post by ElevenLathe »

kenjackson wrote: post #55
ElevenLathe wrote: post #52

Earlier quoted context omitted.

Is this a certainty? I thought it was an open question whether quantum effects are at play in the brain, and those have a counterintuitive relationship with time (to vastly dumb things down in a way my grug mind can comprehend).

Well there’s no evidence of this that I’ve seen. If so, then maybe that is what is the blocker for AGI.

I think it's more that there isn't yet evidence against it. In other words, we're not sure or not if the brain has some kind of special sauce that doesn't just reduce to linear algebra.

Re: I got the highest score on ARC-AGI again swapping Python for English

#75

Post by artrockalter »

0x20cowboy wrote: post #13

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

> It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

True, and "Agentic Workflows" are now playing the same role as "Agile" in that both take the idea that if you have many people/LLMs that can solve toy problems but not real ones then you can still succeed by breaking down the real problems into toy problems and assigning them out.

Re: I got the highest score on ARC-AGI again swapping Python for English

#76

Post by freejazz »

ACCount37 wrote: post #29
HighGoldstein wrote: post #26

Earlier quoted context omitted.

> There is very little practical difference between "understanding" and "reasoning" implemented in human mind and that implemented in LLMs. Source?

The primary source is: measured LLM performance on once-human-exclusive tasks - such as high end natural language processing or commonsense reasoning. Those things were once thought to require a human mind - clearly, not anymore. Human commonsense knowledge can be both captured and applied by a learning algorithm trained on nothing but a boatload of text. But another important source is: loads and loads of mech inter…

I haven't seen LLMs perform common sense reasoning. Feel free to share some links. Your post reads like anthropomorphized nonsense.

Re: I got the highest score on ARC-AGI again swapping Python for English

#78

Post by Workaccount2 »

mjr00 wrote: post #68
naasking wrote: post #60

Earlier quoted context omitted.

> are they really learning to reason, or are they just learning to pattern match to steer generation in the direction of problem-specific reasoning steps that they had been trained on? Are you sure there's a real difference? Do you have a definition of "reasoning" that excludes this?

It's trivial to demonstrate that LLMs are pattern matching rather than reasoning. A good way is to provide modified riddles-that-aren't. As an example: > Prompt: A man working at some white collar job gets an interview scheduled with an MBA candidate. The man says "I can't interview this candidate, he's my son." How is this possible? > ChatGPT: Because the interviewer is the candidate’s mother. (The riddle plays on t…

Over fitting isn't evidence of non-reasoning, but that aside, what's interesting is that ChatGPT (free) trips on this, as did older models. But GPT-5 thinking, Opus 4, and Gemini 2.5 Pro all pointed out that there is no trick and it's likely the man just views it as a conflict of interest to interview his son.

It's hard to say whether this has been trained out (it's an old example) or if it's just another hurdle that general model progression has overcome.

Re: I got the highest score on ARC-AGI again swapping Python for English

#80

Post by freejazz »

kenjackson wrote: post #47
Bjartr wrote: post #44

Earlier quoted context omitted.

> like we all do Do we though? Sure, we communicate sequentially, but that doesn't mean that our internal effort is piecewise and linear. A modern transformer LLM however is. Each token is sampled from a population exclusively dependent on the tokens that came before it. Mechanistically speaking, it works similarly to autocomplete, but at a very different scale. Now how much of an unavoidable handicap this incurs, if…

Our thinking is also based only on events that occurred previously in time. We don’t use events in the future.

This is unhelpfully obtuse
Post reply on HN