Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Seven replies to the viral Apple reasoning paper and why they fall short

garymarcus.substack.com

Post reply on HN
Print view View original

251–260 of 331 posts

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#251

Post by lossolo »

Jabrov wrote: post #35
labrador wrote: post #31

The key insight is that LLMs can 'reason' when they've seen similar solutions in training data, but this breaks down on truly novel problems. This isn't reasoning exactly, but close enough to be useful in many circumstances. Repeating solutions on demand can be handy, just like repeating facts on demand is handy. Marcus gets this right technically but focuses too much on emotional arguments rather than clear explanat…

I’m so tired of hearing this be repeated, like the whole “LLMs are _just_ parrots” thing. It’s patently obvious to me that LLMs can reason and solve novel problems not in their training data. You can test this out in so many ways, and there’s so many examples out there. ______________ Edit for responders, instead of replying to each: We obviously have to define what we mean by "reasoning" and "solving novel problems"…

Your entire edit essentially walks back your earlier strong claims.

None of your current points actually support your position.

1. No, it doesn't. That's a ridiculous claim. Are you seriously suggesting that statistics require reasoning?

2. If you map that language to tokens, it's obvious the model will follow that mapping.

etc.

Here are papers showing that these models can't reason:

https://arxiv.org/abs/2311.00871

https://arxiv.org/abs/2309.13638

https://arxiv.org/abs/2311.09247

https://arxiv.org/abs/2305.18654

https://arxiv.org/abs/2309.01809

You're mistaking pattern matching and the modeling of relationships in latent space for genuine reasoning.

I don't know what you're working on, but while I'm not curing cancer, I am solving problems that aren't in the training data and can't be found on Google. Just a few days ago, Gemini 2.5 Pro literally told me it didn’t know what to do and asked me for help. The other models hallucinated incorrect answers. I solved the problem in 15 minutes.

If you're working on yet another CRUD app, and you've never implemented transformers yourself or understood how they work internally, then I understand why LLMs might seem like magic to you.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#252

Post by dfawcus »

DiogenesKynikos wrote: post #238
xienze wrote: post #236

Earlier quoted context omitted.

But the Tower of Hanoi can be solved without "tools" by humans, simply by understanding the problem, thinking about the solution, and writing it out. Having the LLM shell out to a Python example that it "wrote" (or rather, "pasted" since surely a Python solution to the Tower of Hanoi was part of its training set) is akin to a human Googling "program to solve Tower of Hanoi", copy-pasting and running the solution. Yes…

LLMs are perfectly capable of writing code to solve problems that are not in their training set. I ask LLMs to write code for niche problems that you won't find answers to just by Googling all the time. The LLMs usually get it right.

Maybe they can, but what the human is able to do is examine the tower of Hanoi problem, and the derive the general rule for solving (odd or even number of disks).

Based upon that comprehension, we then need little working memory (tokens) to solve the problem, it just becomes tedious to execute the algorithm.. But the algorithm was derived after considering the first 3 or 4 cases.

Whereas for the moment, LLMS are just pattern matching; whereas we do the pattern match, then derive the generalised rule.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#253

Post by diego898 »

heavyset_go wrote: post #157
jazzypants wrote: post #151

Earlier quoted context omitted.

They're referring to the founder of that website, Eliezer Yudkowsky, who is controversial due to his 2023 Time article that called for a complete halt on the development of AI. https://en.m.wikipedia.org/wiki/Eliezer_Yudkowsky https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

Yudkowsky is controversial for much more than an article from 2023. Yudkowsky lacks credentials and MIRI and its adjacents have proven to be incestuous organizations when it comes to the rationalist cottage industry, one that has a serious problem with sexual abuse and literal cults.

Do you have any references you can share for sexual abuse and/or cults associated with rationalists?

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#254

Post by DiogenesKynikos »

xienze wrote: post #239
DiogenesKynikos wrote: post #238

Earlier quoted context omitted.

LLMs are perfectly capable of writing code to solve problems that are not in their training set. I ask LLMs to write code for niche problems that you won't find answers to just by Googling all the time. The LLMs usually get it right.

> LLMs are perfectly capable of writing code to solve problems that are not in their training set. Examples of these problems? You'll probably find that they're simply compositions of things already in the training set. For example, you might think that "here's a class containing an ID field and foobar field. Make a linked list class that stores inserted items in reverse foobar order with the ID field breaking ties"…

What you're describing is successful generalization from the training dataset, also called "understanding" by laypeople.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#255

Post by Workaccount2 »

sigotirandolas wrote: post #244
Workaccount2 wrote: post #160

Earlier quoted context omitted.

LLMs are (suspected) a few TB in size. Gemma 2 27B, one of the top ranked open source models, is ~60GB in size. LLama 405B is about 1TB. Mind you that they train on likely exabytes of data. That alone should be a strong indication that there is a lot more than memory going on here.

I'm not convinced by this argument. You can fit a bunch of books covering up to MSc level maths on less than 100MB. After that point, more books will mostly be redundant information so it doesn't need much more space for maths beyond that. Similarly TBs of Twitter/Reddit/HN add near zero new information per comment. If anything you can fit an enormous amount of information in 1MB - we just don't need to do it because…

People aren't claiming that they are holding textbooks in their model, that would just be even more evidence of reasoning (the LLM would have to reason what textbook to reference, and then extrapolate from the textbook(s) how to solve the problem at hand - pretty much what students in school do; study the textbook and reason from it to answer new test questions)

People are claiming that the models sit on a vast archive of every answer to every question. i.e. when you ask it 92384 x 333243 = ?, the model is just pulling from where it has seen that before. Anything else would necessitate some level of reasoning.

Also in my own experience, people are stunned when they learn that the models are not exabytes in size.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#256

Post by Workaccount2 »

xienze wrote: post #239
DiogenesKynikos wrote: post #238

Earlier quoted context omitted.

LLMs are perfectly capable of writing code to solve problems that are not in their training set. I ask LLMs to write code for niche problems that you won't find answers to just by Googling all the time. The LLMs usually get it right.

> LLMs are perfectly capable of writing code to solve problems that are not in their training set. Examples of these problems? You'll probably find that they're simply compositions of things already in the training set. For example, you might think that "here's a class containing an ID field and foobar field. Make a linked list class that stores inserted items in reverse foobar order with the ID field breaking ties"…

The problem with this is that anything presented can be claimed to be in the training set, which is likely a zetebyte in size if not larger. However the counter-factual, the LLM failing a problem that is provably in it's training set (there are many), seems to carry no weight.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#257

Post by saberience »

dfawcus wrote: post #252
DiogenesKynikos wrote: post #238

Earlier quoted context omitted.

LLMs are perfectly capable of writing code to solve problems that are not in their training set. I ask LLMs to write code for niche problems that you won't find answers to just by Googling all the time. The LLMs usually get it right.

Maybe they can, but what the human is able to do is examine the tower of Hanoi problem, and the derive the general rule for solving (odd or even number of disks). Based upon that comprehension, we then need little working memory (tokens) to solve the problem, it just becomes tedious to execute the algorithm.. But the algorithm was derived after considering the first 3 or 4 cases. Whereas for the moment, LLMS are just…

LLMs do that too though lol.

The Tower of Hanoi problem is terrible example for somehow suggesting humans are superior.

Firstly, there are plenty of humans who can’t solve this problem even for 3 disks, let alone 6 or 7. Secondly, LLMs can both give you general instructions to solve for any case and they can write out exhaustive move lists too.

Anyway, the fact that there are humans who cannot do Tower of Hanoi already rules it out as a good test of general intelligence anyway. We don’t say that a human doesn’t have “general intelligence” if they cannot solve Towers of Hanoi, so why then would it be a good test for LLM general intelligence?

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#258

Post by procgen »

xienze wrote: post #239
DiogenesKynikos wrote: post #238

Earlier quoted context omitted.

LLMs are perfectly capable of writing code to solve problems that are not in their training set. I ask LLMs to write code for niche problems that you won't find answers to just by Googling all the time. The LLMs usually get it right.

> LLMs are perfectly capable of writing code to solve problems that are not in their training set. Examples of these problems? You'll probably find that they're simply compositions of things already in the training set. For example, you might think that "here's a class containing an ID field and foobar field. Make a linked list class that stores inserted items in reverse foobar order with the ID field breaking ties"…

> that they're simply compositions of things already in the training set

Yes, knowledge is compositional. This is just as true for humans as it is for machines.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#259

Post by saberience »

xienze wrote: post #239
DiogenesKynikos wrote: post #238

Earlier quoted context omitted.

LLMs are perfectly capable of writing code to solve problems that are not in their training set. I ask LLMs to write code for niche problems that you won't find answers to just by Googling all the time. The LLMs usually get it right.

> LLMs are perfectly capable of writing code to solve problems that are not in their training set. Examples of these problems? You'll probably find that they're simply compositions of things already in the training set. For example, you might think that "here's a class containing an ID field and foobar field. Make a linked list class that stores inserted items in reverse foobar order with the ID field breaking ties"…

That’s exactly what humans do though lol.

We reason about things based on our training data. We have a hard time or impossible time reasoning about things we haven’t trained on.

Ie: a human with no experience of board games cannot reason about chess moves. A human with no math knowledge cannot reason about math problems.

How would expect an LLM to reason about something with no training data?

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#260

Post by YeGoblynQueenne »

saberience wrote: post #259
xienze wrote: post #239

Earlier quoted context omitted.

> LLMs are perfectly capable of writing code to solve problems that are not in their training set. Examples of these problems? You'll probably find that they're simply compositions of things already in the training set. For example, you might think that "here's a class containing an ID field and foobar field. Make a linked list class that stores inserted items in reverse foobar order with the ID field breaking ties"…

That’s exactly what humans do though lol. We reason about things based on our training data. We have a hard time or impossible time reasoning about things we haven’t trained on. Ie: a human with no experience of board games cannot reason about chess moves. A human with no math knowledge cannot reason about math problems. How would expect an LLM to reason about something with no training data?

>> Ie: a human with no experience of board games cannot reason about chess moves. A human with no math knowledge cannot reason about math problems.

Then how did the first humans solve math and chess problems, if there were none around solved to give them examples of how to solve them in the first place?

Post reply on HN