Live data from Hacker News

Understanding Reasoning LLMs

magazine.sebastianraschka.com

141–150 of 196 posts

Re: Understanding Reasoning LLMs

#141
post #120

Earlier quoted context omitted.

That doesn't really settle it, just dismiss the question. The submarine analogy could be interpreted to support either conclusion.

Wasn’t the point that process does not matter if we can’t distinguish the end results?

You might be conflating the epistemological point with Turing's test, et cetera. I could not agree more that indistinguishability is a key metric. These days, it is quite possible (at least for me) to distinguish LLM outputs from those of a thinking human, but in the future that could change. Whether LLMs "think" is not an interesting question because these are algorithms, people. Algorithms do not think.

Re: Understanding Reasoning LLMs

#142

Is there any work being done in training LLMs on more restricted formal languages? Something like a constraint solver or automated theorem prover, but much lower level. Specifically something that isn't natural language. That's the only path I could see towards reasoning models being truly effective I know there is work being done with e.g. Lean integration with ChatGPT, but that's not what I mean exactly -- there's…

If I understand your idea correctly, I don't think a "pure" LLM would derive much advantage from this. Sure, you can constrain them to generate something syntactically valid, but there's no way to make them generate something semantically valid 100% of the time. I've seen frontier models muck up their function calling JSON more than once. As long as you're using something statistical like transformers, you're going t…

I wholeheartedly disagree. Logic is inherently statistical due to the very nature of empirical sampling, which is the only method we have for verification. We will eventually find that it's classical, non-statistical logic which was the (useful) approximation/hack, and that statistical reasoning is a lot more "pure" and robust of an approach.

I went into a little more detail here last week: https://news.ycombinator.com/item?id=42871894

> My personal insight is that "reasoning" is simply the application of a probabilistic reasoning manifold on an input in order to transform it into constrained output that serves the stability or evolution of a system.

> This manifold is constructed via learning a decontextualized pattern space on a given set of inputs. Given the inherent probabilistic nature of sampling, true reasoning is expressed in terms of probabilities, not axioms. It may be possible to discover axioms by locating fixed points or attractors on the manifold, but ultimately you're looking at a probabilistic manifold constructed from your input set.

I've been writing and working on this problem a lot over the last few months and hopefully will have something more formal and actionable to share eventually. Right now I'm at the, "okay, this is evident and internally consistent, but what can we actually do with it that other techniques can't already accomplish?" phase that a lot of these metacognitive theories get stuck on.

Re: Understanding Reasoning LLMs

#143

I think the next big problem we will run into with these line of reasoning models is "over-thinking" you can already start to see it. Thinking harder is not the universal pareto improvement everyone seems to think it is. (I understand the irony of using think 4 times here haha)

100%

I do philosophy and it will take an exaggeration I give it, and call it fact.

The non reasoning models will call me out. lol

Re: Understanding Reasoning LLMs

#144

One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular. A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc. I find that all of…

Humans and other animals with cognition have the ability to form theories about the minds of others and can anticipate their reactions. I don’t know if vector spaces and transformers can encode that ability. It’s a key skill in thinking and writing. I definitely tailor my writing for my audience in order to get a point across. Often the goal isn't simply an answer, it’s a convincing answer. Update : forgot a word

They definitely can.

I rolled out reasoning for my interactive reader app, and I tried to extract R1's reasoning traces to use with my existing models, but found its COT for writing wasn't particularly useful*.

Instead of leaning on R1 I came up with my own framework for getting the LLM to infer the reader's underlying frame of mind through long chains of thought, and with enough guidance and a some hand edited examples I was able to get reasoning traces that demonstrated real insight into reader behavior.

Obviously it's much easier in my case because it's an interactive experience: the reader is telling the AI what action they'd like the main character to try, and that in turn is an obvious hint into how they want things go otherwise. But readers don't want everything to go perfectly every time, so it matters that the LLMs are also getting very good picking up on non-obvious signals in reader behavior.

With COT the model infers the reader expectations and state of mind in its own way and then "thinks" itself into how to subvert their expectations, especially in ways that will have a meaningful payoff for the specific reader. That's a huge improvement over an LLM's typical attempts at subversion which tend to bounce between being too repetitive to feel surprising, or too unpredictable to feel rewarding.

(* I agree that current reasoning oriented post-training over-indexes on math and coding, mostly because the reward functions are easier. But I'm also very ok with that as someone trying to compete in the space)

Re: Understanding Reasoning LLMs

#145
Haven't we seen real life examples of this occurring in AI for medical imaging? Models trained on images of tumors state that tumors circled in purple ink or images of tumors that also include a visual scale are over identified as cancerous because they reason that both of those items indicate cancer due to the training data leading them that way?

Re: Understanding Reasoning LLMs

#146
post #25
post #19

Nice article. >Whether and how an LLM actually "thinks" is a separate discussion. The "whether" is hardly a discussion at all. Or, at least one that was settled long ago. "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." --Edsger Dijkstra

The document that quote comes from is hardly a definitive discussion of the topic. “[…] it tends to divert the research effort into directions in which science can not—and hence should not try to—contribute.” is a pretty myopic take. -- http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD898.PDF

Dijkstra is clearly approaching the subject from an engineer/scientist more practical pov. His focus is on the application of the technology to solve problems, from that pov whether AI fits the definition of "human thinking" is indeed uninteresting.

Re: Understanding Reasoning LLMs

#147

One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular. A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc. I find that all of…

> things that aren't well-defined If it's not well defined then you can't do RL on it because without a clear cut reward function the model will learn to do some nonsense instead, simple as.

Well, but: Humans learn to do things well that don't have clear-cut reward functions. Picasso didn't become Picasso because of simple incentives.

So, I question the hypothesis.

Re: Understanding Reasoning LLMs

#148

Earlier quoted context omitted.

But it has to emit hundreds of tokens per test. Does that mean it takes hundreds of times longer to train? Or longer because I imagine the feedback loop can cause huge instabilities in gradients. Or are all GPTs trained on longer formats now; i.e. is "next word prediction" just a basic thing from the beginning of the transformers era?

takes a long time yes, but not longer than pretraining. sparse rewards are a common issue in RL and addressed by many techniques (I'm not expert so I can't say more). Model only does next word prediction and generates a number of trajectories, the correct ones get rewarded (those predictions in the correct trajectory have their gradients propagated back and reinforced).

Good point, hadn't considered that all RL models have the same challenge. So far I've only tinkered with next token prediction and image classification. Now I'm curious to dig more into RL and see how they scale it. Especially without a human in the loop, seems like a challenge to grade the output; it's all wrong wrong wrong random tokens until the model magically guesses the right answer once a zillion years from now.

Re: Understanding Reasoning LLMs

#149
post #87

But how on earth do you train it? With regular LLMs, you get feedback on each word / token you generate, as you can match against training text. With these, you've got to generate hundreds of tokens in the thinking block fiest, and even after that, there's no "matching" next word, only a full solution. And it's either right or wrong, no probabilities to do a gradient on.

There are two RL approaches - process reward models (PRM) that provide feedback on each step of the reasoning chain, and outcome reward models (ORM) that only provide feedback on the complete chain. DeepSeek use an outcome model, and mention some of the difficulties of PRM, including both identifying an individual step as well as how to verify it. The trained reward model provides the gradient.

Re: Understanding Reasoning LLMs

#150

One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular. A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc. I find that all of…

I think the emphasis on coding/math is just because those are the low hanging fruit - they are relatively easy to provide reasoning verification for, both for training purposes and for benchmark scoring. The fact that you can then brag about how good your model is at math, which seems like a high intelligence activity (at least when done by a human) doesn't hurt either!

Reasoning verification in the general case is harder - it seems "LLM as judge" (ask an LLM if it sounds right!) seems to be the general solution.

Post reply on HN