Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs don't do formal reasoning

garymarcus.substack.com

Post reply on HN
Print view View original

91–100 of 128 posts

Re: LLMs don't do formal reasoning

#91

Post by hackinthebochs »

AlienRobot wrote: post #61
hackinthebochs wrote: post #16

Getting tired of seeing this guy's bad arguments get signal boosted. I posted this comment on another LLM thread on the front page today, and I'll just repost it here: LLMs aren't totally out of scope of mathematical reasoning. LLMs roughly do two things, move data around, and recognize patterns. Reasoning leans heavily on moving data around according to context-sensitive rules. This is well within the scope of LLMs.…

What is moving data round? Isn't everything in a computer moving data around? Do you mean backpropagation or somtehing more specific?

Computation generally is partly moving data around, yes. What transformers do is learn how to move data around in a context-relevant manner. This greatly increases the expressivity of the kinds of computations they can perform over traditional deep nets.

https://lilianweng.github.io/posts/2018-06-24-attention/

https://transformer-circuits.pub/2022/in-context-learning-an...

https://transformer-circuits.pub/2021/framework/index.html#r...

Re: LLMs don't do formal reasoning

#92

Post by major4x »

What an obvious article. But it, because it comes from Apple, everybody pays attention. Proof by pedigree. OK, here is my two cents. Firstly, I did my Ph.D. in AI (algorithm design with application to AI) and I also spent seven years applying some of the ideas at Xerox PARC (yes, the same (in)famous research lab). So, I went to and published at many AI conferences (AAAI, ECAI, etc.). Of course, when I was younger and less cynical, I would enter into lengthy philosophical discussions with dignitaries of AI on what does AI mean and it would be long dinners and drinks, and wheelbarrows of ego. Long story, short, there is no such thing as AI. It is a collection of disciplines: the recently famous Machine Learning (transformers trained on large corpora of text), constraint-based reasoning, Boolean satisfiability, theorem proving, probabilistic reasoning, etc., etc. Of course, LLMs are a great achievement and they have good application to Natural Language Processing (also intermingled discipline and considered constituent of AI).

Look at the algorithmic tools used in ML and automated theorem proving for example: ML uses gradient descent (and related numerical methods) for local optimization, while constraint satisfaction/optimization/Boolean satisfiability, SAT modulo-theories, Quantified Boolean Optimization, etc., rely on combinatorial optimization. Mathematically, combinatorial optimization is far more problematic compared to numerical methods and much more difficult, largely because modern computers and NVidia gaming cards are really fast in crunching floating point numbers and also largely that most problems in combinatorial optimization NP-hard or harder.

Now thing of what LLM and local optimization is doing: it is essentially searching/combining sequences of words from Wikipedia and books. But search is not necessarily a difficult problem, it is actually an O(1) problem. While multiplying numbers is an O(n^2.8 (or whatever constant they came up with)) problem while factorization is (God knows what class of complexity) when you take quantum computing into the game).

Great, these are my 2 cents for the day, good luck to the OpenAI investors (I am also investing there a bit as a Bay Area citizen). You guys will certainly make help desk support cheaper...

Re: LLMs don't do formal reasoning

#93

Post by shprd »

whiplash451 wrote: post #21

I am not sure who the target audience of Gary Marcus is. Those who know about LLMs are aware that they do not reason, but also know it not very useful to repeat it over and over again and focus on other aspects of research. Those who don't know about LLMs simply learn to use them in a way that's useful in their life.

If only that was the case. Laywers used them in courts, key people are using it to analyze reports and make decisions for them because it's "AI" and advertised as better than humans. The problem is LLMs output look coherent and make sense so with the advertising, people are misled about what it does and what is capable of.

People are only hearing about AI, how it's revolutionary, and how it's master in every field.

It can solve questions better than me so why would I not use it to help me with everything that I can't figure out?

There are billions spent in marketing to make people buy these products. No one is telling customers to figure it out and see if it's useful.

Even many technical people started getting lost:

  you know what? maybe it does reason. I asked it this novel trick question and it answered correctly. This is a new model, we don't fully understand its capabilities yet.
You might be able to spot little "mistakes" and "exaggeration" and see they're just selling it but people accumulate those "exaggeration" from here and there and build on them collectively.

Re: LLMs don't do formal reasoning

#94

Post by ajross »

adamc wrote: post #48
ajross wrote: post #13

It seems like the needle is now swinging too far back, pointing to "LLMs will NEVER work". And I don't think that's very grounded either. All these criticisms are valid for human beings too . That kind of question trickery trips up school kids all the time. It's hard to use our brains to reason. It takes practice, and the respresentation of the "reasoning" always ends up being alien to our actual cognitive experience…

I struggle to see the use of this comment. Many human beings have jobs where they reason about problems far more complex than this every day. Sure, not every human is great at this. But the interest in using LLMs as agents does kind of require that they can routinely get this right -- the author of the blog post mentions this explicitly.

> Many human beings have jobs where they reason about problems far more complex than this every day.

And they hold degrees from decades of education that taught them how to do that. Kids, even smart ones, can't do this reliably. I have two.

I'm just saying that 3 years into the AI Revolution is a bit premature to demand that they "routinely get this right" when you yourself took probably 20 years to get to that point.

To be blunter: this discourse has a very I Am Very Smart vibe to it, which seems pretty amazingly ironic.

Re: LLMs don't do formal reasoning

#95

Post by hackinthebochs »

bumby wrote: post #58
hackinthebochs wrote: post #16

Getting tired of seeing this guy's bad arguments get signal boosted. I posted this comment on another LLM thread on the front page today, and I'll just repost it here: LLMs aren't totally out of scope of mathematical reasoning. LLMs roughly do two things, move data around, and recognize patterns. Reasoning leans heavily on moving data around according to context-sensitive rules. This is well within the scope of LLMs.…

> The problem is that general problem solving requires potentially arbitrary amounts of moving data Can you expand on this thought?

Forget about solving practical problems for a second. We can just ask the LLM to simulate some arbitrary computation within its context window. But we can in principle require that the output depends on state from arbitrarily many steps in the past. You then need to "carry forward" the required data or otherwise make it available. This is what I mean by moving data. The required associations between data can extend beyond the buffer or available state.

Re: LLMs don't do formal reasoning

#96

Post by Scarblac »

ak_111 wrote: post #7
Dig1t wrote: post #4

You could substitute "LLMs" -> "Humans" and the statement would also be true.

Are you suggesting Humans can't do formal reasoning? Because you can easily teach a four year old not to make illegal moves in chess with very little instructions, and by 10 geniuses like Terence Tao were discussing open math problems with Erdos. If anything this article adds further evidence that whatever the architecture of the human brain it is very different to an LLM architecture.

He said most humans. Comparing the LLM to Terence Tao is saying they're already better than almost every human.

Re: LLMs don't do formal reasoning

#97

Post by bbor »

bartread wrote: post #86
bbor wrote: post #37

Earlier quoted context omitted.

To be fair, and in case it isn’t obvious: this is kinda this guy’s whole schtick. And has been for decades: The inability of standard neural network architectures to reliably extrapolate — and reason formally — has been the central theme of my own work back to 1998 and 2001, and has been a theme in all of my challenges to deep learning, going back to 2012, and LLMs in 2019. Basically he sees his role in human develop…

Thank you: this is really helpful context that, in the case of this author, I wasn't aware of. To be honest I didn't even look at his name, I just skimmed the piece and had that sort of, "oh, it's this all over again," reaction. > When he started connectionism was the underdog camp, and he’s lived to see it take over AI to such an extreme extent that most laypeople would honestly say that AI didn’t exist until, like,…

Yup, well said. If you think we engineers have it tough, think of the poor AI professors — the go-to book for AI survey courses (Russel and Norvig) is maybe 75% symbolic approaches, and students are surely up in arms about that now that everyone’s talking about LLMs.

I spend a lot of time “defending” AI, and I do enjoy pointing out that basically any computer program of any kind is AI, including websites. We don’t even have a good definition of intelligence for people, it’s pure hubris to try to put a solid one onto computers!

Of course, the old (90s?) adage holds true, and should be plastered on billboards across SV, IMO: “AI is whatever hasn't been done yet.” - Larry Tesler https://en.wikipedia.org/wiki/AI_effect

Re: LLMs don't do formal reasoning

#98

Post by wkat4242 »

anon291 wrote: post #52
lifeisstillgood wrote: post #17

Earlier quoted context omitted.

Wait what ? Is that real?

We really really really need to disambiguate the LLM, which is a fixed length, fixed compute time process which takes in an input and produces a token distribution, from the AI system, which takes the output of the LLM and eventually produces something for the user. In this case, all LLMs are fixed-length, but not all AI systems are. An LLM on its own is useless. Current SoTA research includes inserting 'pause' token…

Yes. AIs come in all sorts of flavours.

I think the main thing that happened with LLMs was that people anthropomorphise them because they finally understand what's going on. Other AIs might be smarter by solving complicated mathematical problems but most people don't speak that language so they're not impressed.

LLM vendors should really make this clear but they don't because a magical thinking machine sells well.

Re: LLMs don't do formal reasoning

#99

Post by wkat4242 »

obmelvin wrote: post #23
wkat4242 wrote: post #12

One of the things that kinda illustrate this for me, is that an LLM always uses the same time to process a prompt of the same length. No matter how complicated the problem is. Obviously the complexity of the problem is not actually taken into account.

the o1 model definitely has a somewhat big variance in how long the task takes depending on what you ask it to do

True the o1 model is the one exception though it's really more of a chain of LLMs. I wouldn't consider it a pure LLM.

Also, o1 still fails at many mathematical tasks which the linked article clarifies.

Re: LLMs don't do formal reasoning

#100

Post by lostmsu »

lifeisstillgood wrote: post #17
wkat4242 wrote: post #12

One of the things that kinda illustrate this for me, is that an LLM always uses the same time to process a prompt of the same length. No matter how complicated the problem is. Obviously the complexity of the problem is not actually taken into account.

Wait what ? Is that real?

No, the processing time depends on the length of generated output.
Post reply on HN