Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models reason well, until they don't

arxiv.org

Post reply on HN
Print view View original

121–130 of 224 posts

Re: Reasoning models reason well, until they don't

#121

Post by rightbyte »

20k wrote: post #90
pistoriusp wrote: post #80

Earlier quoted context omitted.

I saw a meme that I think about fairly often: Great apes have learnt sign language, and communicated with humans, since the 1960's. In all that time they've never asked human questions. They've never tried to learn anything new! The theory is that they don't know that there are entities that know things they don't. I like to think that AI are the great apes of the digital world.

Its worth noting that the idea that great apes have learnt sign language is largely a fabrication by a single person, and nobody has ever been able to replicate this. All the communication has to be interpreted through that individual, and anyone else (including people that speak sign language) have confirmed that they're just making random hand motions in exchange for food They don't have the dexterity to really sig…

I mean dogs can learn a simple sign language?

Re: Reasoning models reason well, until they don't

#122

Post by mannykannot »

ftalbot wrote: post #49
My_Name wrote: post #45

I find that they know what they know fairly well, but if you move beyond that, into what can be reasoned from what they know, they have a profound lack of ability to do that. They are good at repeating their training data, not thinking about it. The problem, I find, is that they then don't stop, or say they don't know (unless explicitly prompted to do so) they just make stuff up and express it with just as much confi…

Every token in a response has an element of randomness to it. This means they’re non-deterministic. Even if you set up something within their training data there is some chance that you could get a nonsense, opposite, and/or dangerous result. The chance of that may be low because of things being set up for it to review its result, but there is no way to make a non-deterministic answer fully bound to solving or reason…

There seems to be more to it than that - in my experience with LLMs, they are good at finding some relevant facts but then quite often present a non-sequitur for a conclusion, and the article's title alone indicates that the problem for LRMs is similar: a sudden fall-off in performance as the task gets more difficult. If the issue was just non-determinism, I would expect the errors to be more evenly distributed, though I suppose one could argue that the sensitivity to non-determinism increases non-linearly.

Re: Reasoning models reason well, until they don't

#123

Post by tomlockwood »

code_martial wrote: post #94
tomlockwood wrote: post #87

Earlier quoted context omitted.

So the answer is a few more trillion?

It’s a worthwhile answer if it can be proven correct because it means that we’ve found a way to create intelligence, even if that way is not very efficient. It’s still one step better than not knowing how to do so.

So we're sending a trillion on faith?

Re: Reasoning models reason well, until they don't

#124

Post by acdha »

anal_reactor wrote: post #39

I'm yet to see a task that AI fails at that bottom 10% of population wouldn't also fail at.

The problem is consistency: AI tools usually produce output which _sounds_ like the top 10% but you have to read it carefully to find the bottom 10% parts. We’re not used to that because human performance isn’t that inconsistent and we use history and social factors: someone’s performance goes down when they’re really drunk, but they rarely show up to work in that state and it’s obvious enough that other people recognize that they shouldn’t be trusted.

Re: Reasoning models reason well, until they don't

#125

Post by code_martial »

tomlockwood wrote: post #123
code_martial wrote: post #94

Earlier quoted context omitted.

It’s a worthwhile answer if it can be proven correct because it means that we’ve found a way to create intelligence, even if that way is not very efficient. It’s still one step better than not knowing how to do so.

So we're sending a trillion on faith?

No, that’s not what I said.

Re: Reasoning models reason well, until they don't

#126

Post by Workaccount2 »

My_Name wrote: post #45

I find that they know what they know fairly well, but if you move beyond that, into what can be reasoned from what they know, they have a profound lack of ability to do that. They are good at repeating their training data, not thinking about it. The problem, I find, is that they then don't stop, or say they don't know (unless explicitly prompted to do so) they just make stuff up and express it with just as much confi…

To be fair, we don't actually know what is and isn't in their training data. So instead we just assign successes to "in the training set" and failures to "not in the training set".

But this is unlikely, because they still can fall over pretty badly on things that are definitely in the training set, and still can have success with things that definitely are not in the training set.

Re: Reasoning models reason well, until they don't

#127

Post by MangoToupe »

pistoriusp wrote: post #80
My_Name wrote: post #45

I find that they know what they know fairly well, but if you move beyond that, into what can be reasoned from what they know, they have a profound lack of ability to do that. They are good at repeating their training data, not thinking about it. The problem, I find, is that they then don't stop, or say they don't know (unless explicitly prompted to do so) they just make stuff up and express it with just as much confi…

I saw a meme that I think about fairly often: Great apes have learnt sign language, and communicated with humans, since the 1960's. In all that time they've never asked human questions. They've never tried to learn anything new! The theory is that they don't know that there are entities that know things they don't. I like to think that AI are the great apes of the digital world.

> The theory is that they don't know that there are entities that know things they don't.

This seems like a rather awkward way of putting it. They may just lack conceptualization or abstraction, making the above statement meaningless.

Re: Reasoning models reason well, until they don't

#128

Post by MangoToupe »

joncrocks wrote: post #99
krapht wrote: post #97

Earlier quoted context omitted.

Citation needed.

https://en.wikipedia.org/wiki/Great_ape_language#Criticism_a... - Not word for word, but certainly casting doubt that apes were ever really communicating in the way that people may have thought.

The way linguists define communication via language? Sure. Let's not drag the rest of humanity into this presumption.

Re: Reasoning models reason well, until they don't

#129

Post by darkwater »

squidbeak wrote: post #119
darkwater wrote: post #114

Earlier quoted context omitted.

But those are still approximations to the actual underlying reality. Because the other option (and yes, it's a dichotomy) is that we already defined and understood every detail of the physics that applies to our universe.

Indeed, that is a dichotomy: a false one. Science is exact without being finished.

So, was Newtonian physics exact already?

Re: Reasoning models reason well, until they don't

#130

Post by dankai »

This is not the only paper that scales reasoning complexity / difficulty.

The CogniLoad benchmark does this as well (in addition to scaling reasoning length and distractor ratio). Requiring the LLM to purely reason based on what is in the context (i.e. not based on the information its pretrained on), it finds that reasoning performance decreases significantly as problems get harder (i.e. require the LLM to hold more information in its hidden state simultaneously), but the bigger challenge for them is length.

https://arxiv.org/abs/2509.18458

Disclaimer: I'm the primary author of CogniLoad so feel free to ask me any questions.

Post reply on HN