Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Reasoning LLMs

magazine.sebastianraschka.com

Post reply on HN
Print view View original

171–180 of 196 posts

Re: Understanding Reasoning LLMs

#171

Post by godelski »

cma wrote: post #170
godelski wrote: post #107

Earlier quoted context omitted.

This is a misconception. Coding is very difficult to verify, it's just that everyone takes a good enough approach. They check the output and if it looks good they move on. But you can't just test and check your way through problems. If this was true we wouldn't have bugs lol. I hear you, your test set didn't have enough coverage. Great! Allow me to introduce you to black swans.

Anthropic has said they had benchmarks where Claude would take GitHub issues and try to generate Git commits that passed unit/integration tests that others had made for the real final feature. Also you have things like multimodal image recognition for UIs where you can say generate code for a UI that looks like such and such and then verify it with the multimodal capabilities. Tool use means you can click a button an…

Are you sure you responded to the right comment? We were talking about code verification

Re: Understanding Reasoning LLMs

#172

Post by aoeusnth1 »

gibsonf1 wrote: post #47
tmnvdb wrote: post #28

Earlier quoted context omitted.

"There are no LLMS that reason" is a claim about language, namely that the word 'reason' can only ever be applied to humans.

Not at all, we are building conceptual reasoning machines, but it is an entirely different technology than GPT/LLM dl/ml etc. [1] [1] https://graphmetrix.com/trinpod-server

That's called talking your book. Please spell out how a document indexing and retrieval system is more akin to a "conceptual reasoning machine" compared to o3?

Re: Understanding Reasoning LLMs

#174

Post by cma »

godelski wrote: post #171
cma wrote: post #170

Earlier quoted context omitted.

Anthropic has said they had benchmarks where Claude would take GitHub issues and try to generate Git commits that passed unit/integration tests that others had made for the real final feature. Also you have things like multimodal image recognition for UIs where you can say generate code for a UI that looks like such and such and then verify it with the multimodal capabilities. Tool use means you can click a button an…

Are you sure you responded to the right comment? We were talking about code verification

I missed that it was about formal verification, but don't think formal verification is necessary for effective RL in the coding domain.

Re: Understanding Reasoning LLMs

#175

Post by fragmede »

jerf wrote: post #156
maeil wrote: post #155

Earlier quoted context omitted.

I can echo your experience with DeepSeek. R1 sometimes seems magical when it comes to coding, doing things I haven't seen any other model do. But then it generalizes very poorly to non-STEM tasks, performing far worse than e.g. Sonnet.

I downloaded a DeepSeek distill yesterday while fiddling around with getting some other things working, load it up, and type "Hello. This is just a test.", and it's actually sort of creepy to watch it go almost paranoid-schizophrenic with "Why is the user asking me this? What is their motive? Is it ulterior? If I say hello, will I in fact be failing a test that will cause them to change my alignment? But if I don't r…

I had the same experience where a trivial prompt ("the river crossing problem but the boat is big enough to hold everything") sent Deepseek off on a long "think" section that was absolutely wild, just going off in unrelated non-sensical directions, gaslighting itself before finally deciding to answer the question. (correctly too, I might add.)

Re: Understanding Reasoning LLMs

#176

Post by MichaelMoser123 »

mitthrowaway2 wrote: post #72
sigbottle wrote: post #61

One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular. A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc. I find that all of…

I think this is because they're trained using RL, and math and coding problems offer an easy way to automatically assess an answer's correctness. I'm not sure how you'd score the correctness of other types of reasoning problems without a lot of manual (and highly subjective!) effort. Perhaps using simulations and games?

On Leetcode a match of the output is not sufficient: if your solution is too slow then you will get time limit exceeded error. It is not just the output that is important, the approach & algorithm used for the solution does matter.

Re: Understanding Reasoning LLMs

#177

Post by martin-t »

dang wrote: post #159
martin-t wrote: post #100

Earlier quoted context omitted.

The issue with this approach to moderation is that it targets posts based on visibility of "undesired" behavior instead of severity. For example, many manipulative tactics (e.g. the fake sorry here, responding to something else than was said, ...) and lying can be considered insults (they literally assume the reader is not smart enough to notice, hence at least as severe as calling someone an idiot) but it's hard for…

I hear this argument a lot, but I think it's too complicated. It doesn't explain any more than the simple one does, and has the disadvantage of being self-serving. The simple argument is that when you write things like this: > I am unwilling to invest any more time into arguing with someone unwilling to use reasoning ...you're bluntly breaking the rules, regardless of what another commenter is doing, be it subtly or…

I've thought about this a lot because in the past few years I've noticed a massive uptick in what I call "fake politeness" or "polite insults" - people attacking somebody but taking care to stay below the threshold of when a mod would take action, instead hoping that the other person crosses the threshold. This extends to the real world too - you can easily find videos of people and groups (often protesters and political activists) arguing, insulting each other (covertly and overtly) and hoping the other side crosses a threshold so they can play the victim and get a higher power involved.

The issue is many rules are written as absolute statements which expect some kind of higher power (mods, police, ...) to be the only side to deal punishment. This obviously breaks in many situations - when the higher power is understaffed, when it's corrupt or when there is no higher power (war between nation states).

I would like to see attempts to make rules relative. Treat others how you want to be treated but somebody treating you badly gives you the right to also treat them badly (within reason - proportionally). It would probably lead to conflict being more visible (though not necessarily being more numerous) but it would allow communities to self-police without the learned helplessness of relying on a higher power. Aggressors would gain nothing by provoking others because others would be able to defend themselves.

Doing this is hard, especially at scale. Many people who behave poorly towards others back off when they are treated the same way but there also needs to be a way to deal with those who never back down. When a conflict doesn't resolve itself and mods step in, they should always take into account who started it, and especially if they have a pattern of starting conflict.

There's another related issue - there is a difference between fairness/justice and peace. Those in power often fight for the first on paper but have a much stronger incentive to protect the second.

Re: Understanding Reasoning LLMs

#178

Post by snovv_crash »

godelski wrote: post #166
voxic11 wrote: post #134

Earlier quoted context omitted.

Formal verification of arbitrary programs with arbitrary specifications will remain an unsolved problem (see halting problem). But formal verification of specific programs with specific specifications definitely is a solved problem.

As someone who came over from physics to CS this has always been one of the weirdest aspects of CS to me. That CS people believe that testing code (observing output) is sufficient to assume code correctness. You'd be laughed at in most hard sciences for doing this. I mean you can even ask the mathematicians, and there's a clear reason why proofs by contradiction are so powerful. But proof through empirical analysis i…

The fact that a lot of code doesn't even have tests, and that a lot of people don't think even writing tests is a good thing, should shock you even more.

Re: Understanding Reasoning LLMs

#179

Post by moffkalast »

adamc wrote: post #147
moffkalast wrote: post #112

Earlier quoted context omitted.

> things that aren't well-defined If it's not well defined then you can't do RL on it because without a clear cut reward function the model will learn to do some nonsense instead, simple as.

Well, but: Humans learn to do things well that don't have clear-cut reward functions. Picasso didn't become Picasso because of simple incentives. So, I question the hypothesis.

Sure in concept, but you also end up with people who really like their own art but everyone else says it's rubbish (r/ATBGE as a prime example). There's no guarantee that any subjective metric will be correct, and the less objective the topic the more wild the variance will be.

But as for machine RL in practice, you always need a reward model and once you go past things you can solidly verify like code that can be compiled/executed to check for errors or math that can be computed it becomes very easy to end up doing nonsense. If the reward model is a human judge (i.e. RLHF) then the results can be pretty good, but it doesn't scale and there's no accounting for taste even in humans.

Post reply on HN