Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Reasoning LLMs

magazine.sebastianraschka.com

Post reply on HN
Print view View original

101–110 of 196 posts

Re: Understanding Reasoning LLMs

#101

Post by kirill5pol »

bglazer wrote: post #83
mitthrowaway2 wrote: post #72

Earlier quoted context omitted.

I think this is because they're trained using RL, and math and coding problems offer an easy way to automatically assess an answer's correctness. I'm not sure how you'd score the correctness of other types of reasoning problems without a lot of manual (and highly subjective!) effort. Perhaps using simulations and games?

Games seem like a really under-explored source of data. It’s an area where humans have an intrinsic motivation to interact with others in dialogue, they can be almost arbitrarily open ended, and there tends to be the kind of clean success/failure end states that RL needs. I’m reminded of the high skill Diplomacy bot that Facebook research built but hasn’t really followed up on.

One of the main authors from that diplomacy bot is the lead for reasoning and O1 at OpenAI

Re: Understanding Reasoning LLMs

#102

Post by sigbottle »

triyambakam wrote: post #78
sigbottle wrote: post #61

One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular. A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc. I find that all of…

I'm not sure I would say overfit. I think that coding and math just have clearly definable objectives and verifiable outcomes to give the model. The soft things you mention are more ambiguous so probably are harder to train for.

Sorry, rereading my own comment I'd like to clarify.

Perhaps time spent thinking isn't a great metric, but just looking at deepseek's logs for example, it's chain of thought for many of these "softer" questions are basically just some aggregate wikipedia article. It'll brush on one concept, then move on, without critically thinking about it.

However, for coding problems, no matter how hard or simple, you can get it to just go around in circles, second guess itself, overthink it. And I think this is is kind of a good thing? The thinking at least feels human. But it doesn't even attempt to do any of that for any "softer" questions, even with a lot of my prompting. The highest I was able to get was 50 seconds, I believe (time isn't exactly the best metric, but I'd rate the intrinsic quality of the CoT lower IMO). Again, when I brought this up to people they suggested that math/logic/programming just intrinsically is harder... I don't buy it at all.

I totally agree that it's harder to train for though. And yes, they are next token predictors, shouldn't be hasty to anthropomorphize, etc. But like.... it actually feels like it's thinking when it's coding! It genuinely backtracks and explores the search space somewhat organically. But it won't afford the same luxury for softer questions is my point.

Re: Understanding Reasoning LLMs

#103

Post by daxfohl »

NitpickLawyer wrote: post #95
daxfohl wrote: post #87

But how on earth do you train it? With regular LLMs, you get feedback on each word / token you generate, as you can match against training text. With these, you've got to generate hundreds of tokens in the thinking block fiest, and even after that, there's no "matching" next word, only a full solution. And it's either right or wrong, no probabilities to do a gradient on.

> only a full solution. And it's either right or wrong, no probabilities to do a gradient on. You could use reward functions that do a lot more complicated stuff than "ground_truth == boxed_answer". You could, for example split the "CoT" in paragraphs, and count how many paragraphs match whatever you consider a "good answer" in whatever topic you're trying to improve. You can use embeddings, or fuzzy string matches,…

But it has to emit hundreds of tokens per test. Does that mean it takes hundreds of times longer to train? Or longer because I imagine the feedback loop can cause huge instabilities in gradients. Or are all GPTs trained on longer formats now; i.e. is "next word prediction" just a basic thing from the beginning of the transformers era?

Re: Understanding Reasoning LLMs

#104

Post by Davidzheng »

daxfohl wrote: post #103
NitpickLawyer wrote: post #95

Earlier quoted context omitted.

> only a full solution. And it's either right or wrong, no probabilities to do a gradient on. You could use reward functions that do a lot more complicated stuff than "ground_truth == boxed_answer". You could, for example split the "CoT" in paragraphs, and count how many paragraphs match whatever you consider a "good answer" in whatever topic you're trying to improve. You can use embeddings, or fuzzy string matches,…

But it has to emit hundreds of tokens per test. Does that mean it takes hundreds of times longer to train? Or longer because I imagine the feedback loop can cause huge instabilities in gradients. Or are all GPTs trained on longer formats now; i.e. is "next word prediction" just a basic thing from the beginning of the transformers era?

takes a long time yes, but not longer than pretraining. sparse rewards are a common issue in RL and addressed by many techniques (I'm not expert so I can't say more). Model only does next word prediction and generates a number of trajectories, the correct ones get rewarded (those predictions in the correct trajectory have their gradients propagated back and reinforced).

Re: Understanding Reasoning LLMs

#105

Post by Terr_ »

vector_spaces wrote: post #4

Is there any work being done in training LLMs on more restricted formal languages? Something like a constraint solver or automated theorem prover, but much lower level. Specifically something that isn't natural language. That's the only path I could see towards reasoning models being truly effective I know there is work being done with e.g. Lean integration with ChatGPT, but that's not what I mean exactly -- there's…

I think that would be a fundamental mismatch. LLMs are statistical and lossy and messy, which is what (paradoxically) permits them to get surprisingly-decent results out of messy problems that draw upon an enormous number and variety of messy examples.

But for a rigorously structured language with formal fixed meaning... Now the the LLM has no advantage anymore, only serious drawbacks and limitations. Save yourself millions of dollars and just write a normal parser, expression evaluator, SAT solver, etc.

You'll get answers faster, using fewer resources, with fewer fundamentally unfixable bugs, and it will actually be able to do math.

Re: Understanding Reasoning LLMs

#106

Post by freilanzer »

gibsonf1 wrote: post #47
tmnvdb wrote: post #28

Earlier quoted context omitted.

"There are no LLMS that reason" is a claim about language, namely that the word 'reason' can only ever be applied to humans.

Not at all, we are building conceptual reasoning machines, but it is an entirely different technology than GPT/LLM dl/ml etc. [1] [1] https://graphmetrix.com/trinpod-server

If LLMs can't reason, then this cannot either - whatever this is supposed to be. Not a good argument. Also, since you're apparently working on that product: 'It is difficult to get a man to understand something when his salary depends on his not understanding it.'

Re: Understanding Reasoning LLMs

#107

Post by godelski »

mitthrowaway2 wrote: post #72
sigbottle wrote: post #61

One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular. A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc. I find that all of…

I think this is because they're trained using RL, and math and coding problems offer an easy way to automatically assess an answer's correctness. I'm not sure how you'd score the correctness of other types of reasoning problems without a lot of manual (and highly subjective!) effort. Perhaps using simulations and games?

This is a misconception. Coding is very difficult to verify, it's just that everyone takes a good enough approach. They check the output and if it looks good they move on. But you can't just test and check your way through problems. If this was true we wouldn't have bugs lol. I hear you, your test set didn't have enough coverage. Great! Allow me to introduce you to black swans.

Re: Understanding Reasoning LLMs

#108

Post by nextaccountic »

colonial wrote: post #92
vector_spaces wrote: post #4

Is there any work being done in training LLMs on more restricted formal languages? Something like a constraint solver or automated theorem prover, but much lower level. Specifically something that isn't natural language. That's the only path I could see towards reasoning models being truly effective I know there is work being done with e.g. Lean integration with ChatGPT, but that's not what I mean exactly -- there's…

If I understand your idea correctly, I don't think a "pure" LLM would derive much advantage from this. Sure, you can constrain them to generate something syntactically valid, but there's no way to make them generate something semantically valid 100% of the time. I've seen frontier models muck up their function calling JSON more than once. As long as you're using something statistical like transformers, you're going t…

> there's no way to make them generate something semantically valid 100% of the time.

You don't need to generate semantically valid reasoning 100% of time for such an approach to be useful. You just need to use semantic data to bias them to follow semantically valid paths more often than not (and sometimes consider using constraint solving on the spot, like offloading into a SMT solver or even incorporating it in the model somehow; it would be nice to have AI models that can combine the strengths of both GPUs and CPUs). And, what's more useful, verify that the reasoning is valid at the end of the train of thought, and if it is not, bail out and attempt something else.

If you see AI as solving an optimization problem (given a question, give a good answer) it's kind of evident that you need to probe the space of ideas in an exploratory fashion, sometimes making unfounded leaps (of the "it was revealed to me in a dream" sort), and in this sense it could even be useful that AI can sometimes hallucinate bullshit. But they need afterwards to come with a good justification for the end result, and if they can't find one they are forced to discard their result (even if it's true). Just like humans often come up with ideas in an irrational, subconscious way, and then proceed to rationalize them. One way to implement this kind of thing is to have the LLM generate code for a theorem prover like Coq or Lean, and then at the end run the code - if the prover rejects the code, the reasoning can't possibly be right, and the AI needs to get back to the drawing board

(Now, if the prover accepts the code, the answer may still be wrong, if the premises were encoded incorrectly - but it would still be a net improvement, specially if people can review the Coq code to spot mistakes)

Re: Understanding Reasoning LLMs

#109

Post by ogrisel »

godelski wrote: post #107
mitthrowaway2 wrote: post #72

Earlier quoted context omitted.

I think this is because they're trained using RL, and math and coding problems offer an easy way to automatically assess an answer's correctness. I'm not sure how you'd score the correctness of other types of reasoning problems without a lot of manual (and highly subjective!) effort. Perhaps using simulations and games?

This is a misconception. Coding is very difficult to verify, it's just that everyone takes a good enough approach. They check the output and if it looks good they move on. But you can't just test and check your way through problems. If this was true we wouldn't have bugs lol. I hear you, your test set didn't have enough coverage. Great! Allow me to introduce you to black swans.

Software Engineering is difficult to verify because it requires dealing with ambiguous understanding of the end-user actual needs / value and subtle trade-offs about code maintainability vs feature coverage vs computational performance.

Algorithmic puzzles, on the other hand, both require reasoning and are easy to verify.

There are other things in coding that are both useful and easy to verify: checking that the generated code follows formatting standards or generating outputs with a specific data schema and so on.

Re: Understanding Reasoning LLMs

#110

Post by aqueueaqueue »

bloomingkales wrote: post #63

About three months ago, I kinda casually suggested to HN that I was using a form of refining to improve my LLMs, which is now being described as "reasoning" in this article and other places. My response a few months ago (Scroll down to my username and read that discussion): https://news.ycombinator.com/item?id=41997727 If only I knew DeepSeek was going to tank the market with something as simple as that lol. Note to…

https://news.ycombinator.com/item?id=42001061 is the link I think....
Post reply on HN