Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models reason well, until they don't

arxiv.org

Post reply on HN
Print view View original

141–150 of 224 posts

Re: Reasoning models reason well, until they don't

#141

Post by weltensturm »

squidbeak wrote: post #120
iq176 wrote: post #98

Earlier quoted context omitted.

Scientific method is the process. Science itself includes the study and compendium of understandings, based on a belief system that includes shared understandings just like mathematics. The foundation of these are philosophical beliefs that we can know and understand these things. For example, on a metaphysical level, if the world around us were a simulation, then science could provide understandings about that simul…

This I'm afraid is rubbish. Scientific proofs categorically don't depend on philosophical beliefs. Reality is measurable and the properties measured don't care about philosophy.

> Reality is measurable

Heisenberg would disagree.

Re: Reasoning models reason well, until they don't

#145

Post by acdha »

anal_reactor wrote: post #136
acdha wrote: post #132

Earlier quoted context omitted.

That’s not inconsistent: your bluffer knows they’re making something up and is using their model of you to construct something they think you’ll believe. Someone who can do that isn’t going to suddenly forget how to count the number of letters in a word.

You're wrong. Counting the number of letters in a word is a significantly more difficult task than lying, both for humans and LLMs. Imagine going to a ghetto and asking people "have you ever lied to someone and had them believe the lie", and ask them to spell "continuously". Children learn to lie before they learn to spell.

> Counting the number of letters in a word is a significantly more difficult task than lying

No, it’s not - you don’t even need to be literate to count symbols - but also consider the complexity of the second task and how many skills each requires: unlike counting letters, lying isn’t simple confabulation and requires a theory of mind and some kind of goal. A child who lies to avoid trouble is doing that because they have enough of a world model to know they are going to get in trouble for something even if they haven’t worked out yet that this is unlikely to work.

Re: Reasoning models reason well, until they don't

#146

Post by igravious »

bob_theslob646 wrote: post #81
egberts1 wrote: post #73

It's simple. Don't ingest more than 40KB at a time into its LLM's RAG pipe and its hallucination goes way, way down. Preferably like not at the start and best not to do more than 40KB at a time at all. That's how I learned how to deal with nftables' 120KB parser_bison.y file by breaking them up into clean sections. All of a sudden, a fully-deterministic LL(1) full semantic pathway of nftables' CLI syntax appears befo…

Why 40kb?

and doesn't it depend on the LLM?

Re: Reasoning models reason well, until they don't

#148

Post by dymk »

usrbinbash wrote: post #107
My_Name wrote: post #45

I find that they know what they know fairly well, but if you move beyond that, into what can be reasoned from what they know, they have a profound lack of ability to do that. They are good at repeating their training data, not thinking about it. The problem, I find, is that they then don't stop, or say they don't know (unless explicitly prompted to do so) they just make stuff up and express it with just as much confi…

> They are good at repeating their training data, not thinking about it. Which shouldn't come as a surprise, considering that this is, at the core of things, what language models do: Generate sequences that are statistically likely according to their training data.

This is too large of an oversimplification of how an LLM works. I hope the meme that they are just next token predictors dies out soon, before it becomes a permanent fixture of incorrect but often stated “common sense”. They’re not Markov chains.

Re: Reasoning models reason well, until they don't

#149

Post by usrbinbash »

alyxya wrote: post #23

The key point the paper seems to make is that existing benchmarks have relatively low complexity on reasoning complexity, so they made a new dataset DeepRD with arbitrarily large reasoning complexity and demonstrated that existing models fail at a complex enough problem. Complexity is defined from the complexity of a graph created by modeling the problem as a graph and determining the traversals needed to go from som…

> I don't think there's evidence that this issue would persist after continuing to scale models to be larger and doing more RL

And how much larger do we need to make the models? 2x? 3x? 10x? 100x? How large do they need to get before scaling-up somehow solves everything?

Because: 2x larger, means 2x more memory and compute required. Double the cost or half the capacity. Would people still pay for this tech if it doubles in price? Bear in mind, much of it is already running at a loss even now.

And what if 2x isn't good enough? Would anyone pay for a 10x larger model? Can we even realistically run such models as anything other than a very expensive PoC and for a very short time? And whos to say that even 10x will finally solve things? What if we need 40x? Or 100x?

Oh, and of course: Larger models also require more data to train them on. And while the Internet is huge, it's still finite. And when things grow geometrically, even `sizeof(internet)` eventually runs out ... and, in fact, may have done so already [1] [2]

What if we actually discover that scaling up doesn't even work at all, because of diminishing returns? Oh wait, looks like we did that already: [3]

[1]: https://observer.com/2024/12/openai-cofounder-ilya-sutskever...

[2]: https://biztechweekly.com/ai-training-data-crisis-how-synthe...

[3]: https://garymarcus.substack.com/p/confirmed-llms-have-indeed...

Re: Reasoning models reason well, until they don't

#150

Post by adastra22 »

dymk wrote: post #148
usrbinbash wrote: post #107

Earlier quoted context omitted.

> They are good at repeating their training data, not thinking about it. Which shouldn't come as a surprise, considering that this is, at the core of things, what language models do: Generate sequences that are statistically likely according to their training data.

This is too large of an oversimplification of how an LLM works. I hope the meme that they are just next token predictors dies out soon, before it becomes a permanent fixture of incorrect but often stated “common sense”. They’re not Markov chains.

They are next token predictors though. That is literally wha they are. Nobody is saying they are simple Markov chains.
Post reply on HN