Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Seven replies to the viral Apple reasoning paper and why they fall short

garymarcus.substack.com

Post reply on HN
Print view View original

271–280 of 331 posts

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#271

Post by godelski »

elbear wrote: post #241
johnecheck wrote: post #231

Earlier quoted context omitted.

Because that would prove absolutely nothing. There are numerous examples of tower of Hanoi explanations in the training set.

How do you check that a human understood it and not simply memorised different approaches?

It's hard. But usually we ask several variations and make them show their work.

But a human also isn't an LLM. It is much harder for them to just memorize a bunch of things, which makes evaluation easier. But they also get tired and hungry, which makes evaluation harder ¯\_(ツ)_/¯

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#272

Post by YeGoblynQueenne »

Jabrov wrote: post #35
labrador wrote: post #31

The key insight is that LLMs can 'reason' when they've seen similar solutions in training data, but this breaks down on truly novel problems. This isn't reasoning exactly, but close enough to be useful in many circumstances. Repeating solutions on demand can be handy, just like repeating facts on demand is handy. Marcus gets this right technically but focuses too much on emotional arguments rather than clear explanat…

I’m so tired of hearing this be repeated, like the whole “LLMs are _just_ parrots” thing. It’s patently obvious to me that LLMs can reason and solve novel problems not in their training data. You can test this out in so many ways, and there’s so many examples out there. ______________ Edit for responders, instead of replying to each: We obviously have to define what we mean by "reasoning" and "solving novel problems"…

>> 1) Next token prediction can itself be argued to be a task that requires reasoning

That is wishful thinking popularised by Ilya Sutskever and Greg Brockman of OpenAI to "explain" why LLMs are a different class of system than smaller language models or other predictive models.

I'm sorry to say that (John Mearsheimer voice) that's simply not a serious argument. Take a multivariate regression model that predicts blood pressure from demographic data (age, sex, weight, etc). You can train a pretty accurate model for that kind of task if you have enough data (a few thousand data points). Does that model need to "reason" about human behaviour in order to be good at predicting BP? Nope. All it needs is a lot of data. That's how statistics works. So why is it different for a predictive model of BP and different for a next-token prediction model? The only answer seems to be "because language is magickal and special". But without any attempt to explain why, in terms of sequence prediction, language is special. Unless the er reasoning is that humans can produce language, humans can reason, LLMs can produce language, therefore LLMs can reason; which obviously doesn't follow.

But I have to guess here because neither Sutskever nor Brockman have ever tried to explain why next token prediction needs reasoning (or, more precisely, "understanding", the term they have used).

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#273

Post by charcircuit »

barrkel wrote: post #235
charcircuit wrote: post #110

Earlier quoted context omitted.

>with tool use A LLM with tool use can solve anything. It is interesting to try and measure its capabilities without tools.

I don't think the first is true at all, unless you imagine some powerful oracle tools. I think the second is interesting for comparing models, but not interesting for determining the limits of what models can automate in practice. It's the prospect of automating labour which makes AI exciting and revolutionary, not their ability when arbitrarily restricted.

Search is already a pretty powerful oracle to defer an answer to a human and is a common tool most AI use today.

What current models can automate is not what the paper was trying to answer.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#274

Post by godelski »

jsnell wrote: post #216
Brystephor wrote: post #178

Earlier quoted context omitted.

Forcing reasoning is analogous to requiring a student to show their work when solving a problem if im understanding the paper correctly. > you’d have to either memorize the entire answer before speaking or come up with a simple pattern you could do while reciting that takes significantly less brainpower This part i dont understand. Why would coming up with an algorithm (e.g. a simple pattern) and reciting it be impos…

The paper doesn't mention it because either the researchers did not care to check the outputs manually, or reporting what was in the outputs would have made it obvious what their motives were. When this research has been reproduced, the "failures" on the Tower of Hanoi are the model printing out a bunch of steps, saying there is no point in doing it thousands of times more. And they they'd either output an the algori…

It's really easy to make a billion dollars. Just make a really useful app and sell it. There's no point explaining the rest since it's so trivial.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#275

Post by godelski »

YeGoblynQueenne wrote: post #272
Jabrov wrote: post #35

Earlier quoted context omitted.

I’m so tired of hearing this be repeated, like the whole “LLMs are _just_ parrots” thing. It’s patently obvious to me that LLMs can reason and solve novel problems not in their training data. You can test this out in so many ways, and there’s so many examples out there. ______________ Edit for responders, instead of replying to each: We obviously have to define what we mean by "reasoning" and "solving novel problems"…

>> 1) Next token prediction can itself be argued to be a task that requires reasoning That is wishful thinking popularised by Ilya Sutskever and Greg Brockman of OpenAI to "explain" why LLMs are a different class of system than smaller language models or other predictive models. I'm sorry to say that (John Mearsheimer voice) that's simply not a serious argument. Take a multivariate regression model that predicts bloo…

  > That is wishful thinking popularised by Ilya Sutskever
Ilya and Hinton have claimed even crazier things

  | to understand next token prediction you must understand the casual reality
This is objectively false. It's a result known in physics to be wrong for centuries. You can probably reason a weaker case yourself, that I'm sure you can make accurate predictions about some things without fully understanding them.

But the stronger version is the entire difficulty of physics and causal modeling. Distinguishing a confounding variable is very very hard. But you can still make accurate predictions without access to the underlying causal graph

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#276

Post by godelski »

socalgal2 wrote: post #206
aucisson_masque wrote: post #67

Earlier quoted context omitted.

> It’s patently obvious that LLMs can reason and solve novel problems not in their training data. Would you care to tell us more ? « It’s patently obvious » is not really an argument, I could say just as well that everyone know LLM can’t resonate or think (in the way we living beings do).

I'm working on new API. I asked the LLM to read the spec and write tests for it. It does. I don't know if that's "reasoning". I know that no tests exist for this API. I know that the internet is not full of training data for this API because it's a new API. It's also not a CRUD API or some other API that's got a common pattern. And yet, with a very short prompt, Gemini Code Assist wrote valid tests for a new feature.…

  > I know that the internet is not full of training data for this API because it's a new API.
1) are you sure? That's a bold guess. It was also a really stupid assumption made by the HumanEval benchmark authors. That if you "hand write" simple leet code style questions then you can train on all of GitHub. Go ahead, go look at what kinds of questions are in that benchmark...

2) LLMs aren't discrete databases. They are curve fitting functions. Compression. They work in very very high dimensions. They can generate new data but that is limited. People mostly aren't saying that LLMs can't create novel things but that they can't reason in the way that humans can. Humans can't memorize half of what a LLM can yet are able to figure out lots of crazy shit.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#277

Post by elbear »

godelski wrote: post #271
elbear wrote: post #241

Earlier quoted context omitted.

How do you check that a human understood it and not simply memorised different approaches?

It's hard. But usually we ask several variations and make them show their work. But a human also isn't an LLM. It is much harder for them to just memorize a bunch of things, which makes evaluation easier. But they also get tired and hungry, which makes evaluation harder ¯\_(ツ)_/¯

If we're talking about solving an equation, for example, it's not hard to memorize. Actually, that's how most students do it, they memorize the steps and what goes where[1].

But they don't really know why the algorithm works the way it does. That's what I meant by understanding.

[1] In learning psychology there is something called the interleaving effect. What it says is that you solve several problems of the same kind, you start to do it automatically after the 2nd or the 3rd problem, so you stop really learning. That's why you should interleave problems that are solved with different approaches/algorithms, so you don't do things on autopilot.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#278

Post by qarl »

zoul wrote: post #200
qarl wrote: post #182

Earlier quoted context omitted.

> The paper doesnt mention the models coming up with the algorithm at all AFAIK. And that's because they specifically hamstrung their tests so that the LLMs were not "allowed" to generate algorithms. If you simply type "Give me the solution for Towers of Hanoi for 12 disks" into chatGPT it will happily give you the answer. It will write program to solve it, and then run that program to produce the answer. But accordi…

This is not about finding the most effective solution, it’s about showing that they “understand” the problem. Could they write the algorithm if it were not in their training set?

That's an interesting question. It's not the one they are trying to answer, however.

From my personal experience: yes, if you describe a problem without mentioning the name of the algorithm, an LLM will detect and apply the algorithm appropriately.

They behave exactly how a smart human would behave. In all cases.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#280

Post by godelski »

elbear wrote: post #277
godelski wrote: post #271

Earlier quoted context omitted.

It's hard. But usually we ask several variations and make them show their work. But a human also isn't an LLM. It is much harder for them to just memorize a bunch of things, which makes evaluation easier. But they also get tired and hungry, which makes evaluation harder ¯\_(ツ)_/¯

If we're talking about solving an equation, for example, it's not hard to memorize. Actually, that's how most students do it, they memorize the steps and what goes where[1]. But they don't really know why the algorithm works the way it does. That's what I meant by understanding. [1] In learning psychology there is something called the interleaving effect. What it says is that you solve several problems of the same ki…

Yes, tests fail in this method. But I think you can understand why the failure is larger when we're talking about a giant compression machine. It's not even a leap in logic. Maybe a small step
Post reply on HN