Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Seven replies to the viral Apple reasoning paper and why they fall short

garymarcus.substack.com

Post reply on HN
Print view View original

161–170 of 331 posts

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#161

Post by ninjin »

jazzypants wrote: post #151
diego898 wrote: post #147

Earlier quoted context omitted.

Sorry - what do you mean by yud-cult? Searching google didn’t help me (as far as I can tell) - I view LW from an outside perspective as well, but don’t understand the reference

They're referring to the founder of that website, Eliezer Yudkowsky, who is controversial due to his 2023 Time article that called for a complete halt on the development of AI. https://en.m.wikipedia.org/wiki/Eliezer_Yudkowsky https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

It was not so much the call for a complete halt that caused controversy, but rather this part of his piece in Time (my emphasis):

"Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that's what it takes to reduce the risk of large AI training runs.

That's the kind of policy change that would cause my partner and I to hold each other, and say to each other that a miracle happened, and now there's a chance that maybe [our daughter] will live."

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#162

Post by mathgradthrow »

FINDarkside wrote: post #108
thomasahle wrote: post #96

> 1. Humans have trouble with complex problems and memory demands. True! But incomplete. We have every right to expect machines to do things we can’t. [...] If we want to get to AGI, we will have to better. I don't get this argument. The paper is about "whether RLLMs can think". If we grant "humans make these mistakes too", but also "we still require this ability in our definition of thinking", aren't we saying "thin…

Agreed. But also his point about AGI is incorrect. AI that will perform on the level of average human in every task is AGI by definition.

the average human is good at something, and sucks at almost everything. Human performance at chess and average performance at chess differ by 7 orders of magnitude.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#163

Post by simonw »

FINDarkside wrote: post #137
simonw wrote: post #116

Earlier quoted context omitted.

That very much depends on which AGI definition you are using. I imagine there are a dozen or so variants out there. See also "AI" and "agents" and (apparently) "vibe coding" and pretty much every other piece of jargon in this field.

I think it's very widely accepted definition and there's really no competing definitions either as far as I know. While some people might think AGI means superintelligence, it's only because they've heard the term but never bothered to look up what it means.

OpenAI: https://openai.com/index/how-should-ai-systems-behave/#citat...

"By AGI, we mean highly autonomous systems that outperform humans at most economically valuable work."

AWS: https://aws.amazon.com/what-is/artificial-general-intelligen...

"Artificial general intelligence (AGI) is a field of theoretical AI research that attempts to create software with human-like intelligence and the ability to self-teach. The aim is for the software to be able to perform tasks that it is not necessarily trained or developed for."

DeepMind: https://arxiv.org/abs/2311.02462

"Artificial General Intelligence (AGI) is an important and sometimes controversial concept in computing research, used to describe an AI system that is at least as capable as a human at most tasks. [...] We argue that any definition of AGI should meet the following six criteria: We emphasize the importance of metacognition, and suggest that an AGI benchmark should include metacognitive tasks such as (1) the ability to learn new skills, (2) the ability to know when to ask for help, and (3) social metacognitive abilities such as those relating to theory of mind. The ability to learn new skills (Chollet, 2019) is essential to generality, since it is infeasible for a system to be optimized for all possible use cases a priori [...]"

The key difference appears to be around self-teaching and meta-cognition. The OpenAI one shortcuts that by focusing on "outperform humans at most economically valuable work", but others make that ability to self-improve key to their definitions.

Note that you said "AI that will perform on the level of average human in every task" - which disagrees very slightly with the OpenAI one (they went with "outperform humans at most economically valuable work"). If you read more of the DeepMind paper it mentions "this definition notably focuses on non-physical tasks", so their version of AGI does not incorporate full robotics.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#164

Post by ninjin »

ramchip wrote: post #155
senko wrote: post #25

Earlier quoted context omitted.

Gary Marcus isn't about "getting real", it's making a name for himself as a contrarian to the popular AI narrative. This article may seem reasonable, but here he's defending a paper that in his previous article he called "A knockout blow for LLMs". Many of his articles seem reasonable (if a bit off) until you read a couple dozen a spot a trend.

I was very put off by his article "A knockout blow for LLMs?", especially all the fuss he was making about using his own name as a verb to mean debunking AI hype...

Marcus comes with a very standard cognitive science criticism of statistical approaches to artificial intelligence, many parts of which dates back to the late 50s from when the field was born and moved to distance itself from behaviourism. The worst part to me is not that his criticism is entirely wrong, but rather that it is obvious and yet peddled as something that those of us that develop statistical approaches are completely ignorant of. To make matters worse, instead of developing alternative approaches (like plenty of my colleagues in cognitive science do!), he simply reiterates pretty much the same points over and over and has done so at least for the last twenty or so years. He and others paint themselves as sceptics and bulwarks against the current hype (which I can assure you, I hate at least as much as they do). But, to me, they are cynics, not sceptics.

I try to maintain a positive and open mind of other researchers, but Marcus lost me pretty much at "first contact" when a student in the group who leaned towards cognitive science had us read "Deep Learning: A Critical Appraisal" by Marcus (2018) [1] back around when it was published. Finally I could get into the mind of this guy so many people were talking about! 27 pages and yet I learned next to nothing new as the criticism was just the same one we have heard for decades: "Statistical learning has limits! It may not lead to 'truly" intelligent machines!". Not only that, the whole piece consistently conflates deep learning and statistical learning for no reason at all, reads as if it was rushed (and not proofed), emphasises the author's research strongly rather than giving a broad overview, etc. In short, it is bad, very bad as a scientific piece. At times, I read short excerpts of an article Marcus has written and yet sadly it is pretty much the same thing all over again.

[1]: https://arxiv.org/abs/1801.00631

There is a horrible market to "sell" hype when it comes to artificial intelligence, but there is also a horrible market to "sell" anti-hype. Sadly, both brings traffic, attention, talk invitations, etc. Two largely unscientific tribes, that I personally would rather do without, with their own profiting gurus.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#165

Post by Jensson »

Workaccount2 wrote: post #154
wohoef wrote: post #52

Good article giving some critique to Apple's paper and Gary Marcus specifically. https://www.lesswrong.com/posts/5uw26uDdFbFQgKzih/beware-gen...

What gets me, and the author talks about it in the post, is that people will readily attribute correct answers to "its in the training set" but nobody says anything about incorrect answers that are in the training set. LLMs get stuff in the training set wrong all the time, but nobody uses it as evidence that it probably can't lean too hard on it's memorization for complex questions it does get right. It puts LLMs in…

> It puts LLMs in an impossible position; if they are right, they memorized it, if they are wrong, they cannot reason.

Both of those can be true at the same time though. They memorize a lot of things, but its fuzzy and when they remember wrong they cannot fix it via reasoning.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#166

Post by Jensson »

Workaccount2 wrote: post #156
goalieca wrote: post #87

Earlier quoted context omitted.

So far they cannot even answer questions which are straight up fact checking and search engine like queries. Reasoning means they would be able to work through a problem and generate a proof they way a student might.

So if they have bad memory, then they must be reasoning to get the correct answer for the problems they do solve?

A clock that is right twice a day is still broken.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#167

Post by jes5199 »

I think the Apple paper is practically a hack job - the problem was set up in such a way that the reasoning models must do all of their reasoning before outputting any of their results. Imagine a human trying to solve something this way: you’d have to either memorize the entire answer before speaking or come up with a simple pattern you could do while reciting that takes significantly less brainpower - and past a certain size/complexity, it would be impossible.

And this isn’t how LLMs are used in practice! Actual agents do a thinking/reasoning cycle after each tool-use call. And I guarantee even these 6-month-old models could do significantly better if a researcher followed best practices.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#168

Post by travisjungroth »

aucisson_masque wrote: post #67
Jabrov wrote: post #35

Earlier quoted context omitted.

I’m so tired of hearing this be repeated, like the whole “LLMs are _just_ parrots” thing. It’s patently obvious to me that LLMs can reason and solve novel problems not in their training data. You can test this out in so many ways, and there’s so many examples out there. ______________ Edit for responders, instead of replying to each: We obviously have to define what we mean by "reasoning" and "solving novel problems"…

> It’s patently obvious that LLMs can reason and solve novel problems not in their training data. Would you care to tell us more ? « It’s patently obvious » is not really an argument, I could say just as well that everyone know LLM can’t resonate or think (in the way we living beings do).

Copied from a past comment of mine:

I just made up this scenario and these words, so I'm sure it wasn't in the training data.

Kwomps can zark but they can't plimf. Ghirns are a lot like Kwomps, but better zarkers. Plyzers have the skills the Ghirns lack.

Quoning, a type of plimfing, was developed in 3985. Zhuning was developed 100 years earlier. I have an erork that needs to be plimfed. Choose one group and one method to do it.

> Use Plyzers and do a Quoning procedure on your erork.

If that doesn't count as reasoning or generalization, I don't know what does.

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#169

Post by bluefirebrand »

FINDarkside wrote: post #137
simonw wrote: post #116

Earlier quoted context omitted.

That very much depends on which AGI definition you are using. I imagine there are a dozen or so variants out there. See also "AI" and "agents" and (apparently) "vibe coding" and pretty much every other piece of jargon in this field.

I think it's very widely accepted definition and there's really no competing definitions either as far as I know. While some people might think AGI means superintelligence, it's only because they've heard the term but never bothered to look up what it means.

Doesn't the "G" in AGI stand for "General" as in "Generally Good at everything"?

Re: Seven replies to the viral Apple reasoning paper and why they fall short

#170

Post by hintymad »

zer00eyz wrote: post #107
hintymad wrote: post #92

Earlier quoted context omitted.

Honest question: does the opinion of Gary Marcus still count? His criticism seems more philosophical than scientific. It's hard for me see what he builds or reasons to get to his conclusions.

> seems more philosophical than scientific I think this is a fair assessment but reason, and intelligence dont really have an established control or control group. If you build a test and say "Its not intelligent because it can't..." and someone goes out and add's that feature in is it suddenly now intelligent? If we make a physics break through tomorrow is there any LLM that is going to retain that knowledge permane…

> The current crop of tech doesn't get us to AGI

I actually agree with this. Time and again, I can see that LLMs do not really understand my questions, let alone being able to perform logical deductions beyond in-distribution answers. What I’m really wondering is whether Marcus’s way of criticizing LLMs is valid.

Post reply on HN