Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Reasoning LLMs

magazine.sebastianraschka.com

Post reply on HN
Print view View original

61–70 of 196 posts

Re: Understanding Reasoning LLMs

#61

Post by sigbottle »

One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular.

A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc.

I find that all of these reasoning models really will overfit and overthink if you attach some level of math problem to it but it will barely think for anything else. I had friends suggest to me (I can't tell if in jest or seriously) that other fields don't require thinking, but I dunno, a lot of these "soft things" I think about really hard and don't have great solutions to.

I've always been a fan of self-learning, for example - wouldn't it be great to have a conversation partner who can both infer and understand your misconceptions about complex topics when trying to learn, just from a few sentences, and then guide you for that?

It's not like it's fundamentally impossible. These LLMs definitely can solve harder coding problems when you make them think. It's just that, I'm pretty sure (and it's really noticable with deepseek) that they're overfit towards coding/math puzzles in particular.

It's really noticable with deepseek when you ask its reasoning model to just write some boilerplate code... you can tell it's completely overfit because it will just overthink and overthink and overthink. But it doesn't do that for example, with "soft" questions. In my opinion, this points to the idea that it's not really deciding for itself "how much thinking is enough thinking" and that it's just really overfit. Which I think can be solved, again, but I think it's more of a training decision issue.

Re: Understanding Reasoning LLMs

#62

Post by _yb2s »

codr7 wrote: post #57
layer8 wrote: post #50

Earlier quoted context omitted.

Magic wasn’t mentioned here. We don’t understand the emerging behavior, in the sense that we can’t reason well about it and make good predictions about it (which would allow us to better control and develop it). This is similar to how understanding chemistry doesn’t imply understanding biology, or understanding how a brain works.

Exactly, we don't understand, but we want to believe it's reasoning, which would be magic.

There's no belief or magic required, the word 'reasoning' is used here to refer to an observed capability, not a particular underlying process.

We also don't understand exactly how humans reason, so any claims that humans are capable of reasoning is also mostly an observation about abilities/capabilities.

Re: Understanding Reasoning LLMs

#63

Post by bloomingkales »

About three months ago, I kinda casually suggested to HN that I was using a form of refining to improve my LLMs, which is now being described as "reasoning" in this article and other places.

My response a few months ago (Scroll down to my username and read that discussion):

https://news.ycombinator.com/item?id=41997727

If only I knew DeepSeek was going to tank the market with something as simple as that lol.

Note to self, take your intuition seriously.

Re: Understanding Reasoning LLMs

#64

Post by _yb2s »

jakefromstatecs wrote: post #51
_yb2s wrote: post #35

Earlier quoted context omitted.

> You don't seem to understand how they work I don't think anyone understands how they work- these type of explanations aren't very complete or accurate. Such explanations/models allow one to reason out what types of things they should be capable of vs incapable of in principle regardless of scale or algorithm tweaks, and those predictions and arguments never match reality and require constant goal post shifting as t…

> I don't think anyone understands how they work Yes we do, we literally built them. > We understand how we brought them about via setting up an optimization problem in a specific way, that isn't the same at all as knowing how they work. You're mistaking "knowing how they work" with "understanding all of the emergent behaviors of them" If I build a physics simulation, then I know how it works. But that's a separate q…

> You're mistaking "knowing how they work" with "understanding all of the emergent behaviors of them"

By knowing how they work I specifically mean understanding the emergent capabilities and behaviors, but I don't see how it is a mistake. If you understood physics but knew nothing about cars, you can't claim to understand how a car works "simple, it's just atoms interacting according to the laws of physics." That would not let you, e.g. explain its engineering principles or capabilities and limitations in any meaningful way.

Re: Understanding Reasoning LLMs

#65

Post by famouswaffles »

HarHarVeryFunny wrote: post #38
scarmig wrote: post #12

Earlier quoted context omitted.

> But I think it's inappropriate to claim that models like R1 are "good at deductive or inductive reasoning" when that is demonstrably not true, they are incapable of even the simplest "out-of-distribution" deductive reasoning: https://xcancel.com/JJitsev/status/1883158738661691878 Your link says that R1, not all models like R1, fails at generalization. Of particular note: > We expose DeepSeek R1 to the variations of…

I'd expect that OpenAI's stronger reasoning models also don't generalize too far outside of the areas they are trained for. At the end of the day these are still just LLMs, trying to predict continuations, and how well they do is going to depend on how well the problem at hand matches their training data. Perhaps the type of RL used to train them also has an effect on generalization, but choice of training data has t…

Nobody generalizes too far outside the areas they're trained for. Probably that length, 'far' is shorter with today's state of the art but the presence of failure modes don't mean anything.

Re: Understanding Reasoning LLMs

#66

Post by danielmarkbruce »

_yb2s wrote: post #60
gsam wrote: post #59

Earlier quoted context omitted.

I don't like wading into this debate when semantics are very personal/subjective. But to me, it seems like almost a sleight of hand to add the stochastic part, when actually they're possibly weighted more on the parrot part. Parrots are much more concrete, whereas the term LLM could refer to the general architecture. The question to me seems: If we expand on this architecture (in some direction, compute, size etc.),…

I'm sure both of you know this, but "stochastic parrot" refers to the title of a research article that contained a particular argument about LLM limitations that had very little to do with parrots.

The term is much more broadly known than the content of that (rather silly) paper.... I'm not even certain that it's the first use of the term.

Re: Understanding Reasoning LLMs

#67

Post by astrange »

jakefromstatecs wrote: post #51
_yb2s wrote: post #35

Earlier quoted context omitted.

> You don't seem to understand how they work I don't think anyone understands how they work- these type of explanations aren't very complete or accurate. Such explanations/models allow one to reason out what types of things they should be capable of vs incapable of in principle regardless of scale or algorithm tweaks, and those predictions and arguments never match reality and require constant goal post shifting as t…

> I don't think anyone understands how they work Yes we do, we literally built them. > We understand how we brought them about via setting up an optimization problem in a specific way, that isn't the same at all as knowing how they work. You're mistaking "knowing how they work" with "understanding all of the emergent behaviors of them" If I build a physics simulation, then I know how it works. But that's a separate q…

We didn't really build them, we do billion-dollar random searches for them in parameter space.

Re: Understanding Reasoning LLMs

#68

Post by danielmarkbruce »

jamiek88 wrote: post #58
danielmarkbruce wrote: post #55

Earlier quoted context omitted.

By knowing what each of the letters stand for

Well that’s disappointing. It was an extraordinary claim that really interested me. Thought I was about to be learn! Instead, I just met an asshole.

When someone says "i'm in the cult that believes X", don't expect a water tight argument for the existence of X.

Re: Understanding Reasoning LLMs

#69

Post by bloomingkales »

sigbottle wrote: post #61

One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular. A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc. I find that all of…

It's a human bias that also exists outside of this current problem space. Take programmers for example, there is a strong bias that is pushed about how mathematically oriented minds are better at programming. This bias has shown up in the training phase of AI, as we believe programming patterns lead to better reasoning (train them on code examples, and then distill the model down, as it now has the magical prowess of a mathematically oriented mind, so they say). When it comes to AI ethics, this is an ethical problem for those that don't think about this stuff. We're seeding these models with our own agenda.

These concepts will be shattered in the long run hopefully, because they are so small.

Re: Understanding Reasoning LLMs

#70

Post by ggm »

gsam wrote: post #59
ggm wrote: post #49

Earlier quoted context omitted.

A good literary production. I would have been proud of it had I thought of it, but it's a path to observe a strong "whataboutery" element that if we use "stochastic parrot" as shorthand and you dislike the term, now you understand why we dislike the constant use of "infer", "reason" and "hallucinate" Parrots are self aware, complex reasoning brains which can solve problems in geometry, tell lies, and act socially or…

I don't like wading into this debate when semantics are very personal/subjective. But to me, it seems like almost a sleight of hand to add the stochastic part, when actually they're possibly weighted more on the parrot part. Parrots are much more concrete, whereas the term LLM could refer to the general architecture. The question to me seems: If we expand on this architecture (in some direction, compute, size etc.),…

"Expand the architecture" .. "get something much more powerful" .. "more dilithium crystals, captain"

Like I said elsewhere in this overall thread, we've been here before. Yes, you do see improvements in larger datasets, weighted models over more inputs. I suggest, I guess I believe (to be more honest) that no amount of "bigger" here will magically produce AGI simply because of the scale effect.

There is no theory behind "more" and that means there is no constructed sense of why, and the absence of abstract inductive reasoning continues to say to me, this stuff isn't making a qualitative leap into emergent anything.

It's just better at being an LLM. Even "show your working " is pointing to complex causal chains, not actual inductive reasoning as I see it.

Post reply on HN