Reasoning models reason well, until they don't

Post by galaxyLogic » Fri, Oct 31, 2025, 7:38 PM UTC

ralfd wrote: post #139

conception wrote: post #100

Earlier quoted context omitted.

Searching for koko ape fraud seems to produce a lot.

> In his lecture, Sapolsky alleges that Patterson spontaneously corrects Koko’s signs: “She would ask, ‘Koko, what do you call this thing?’ and [Koko] would come up with a completely wrong sign, and Patterson would say, ‘Oh, stop kidding around!’ And then Patterson would show her the next one, and Koko would get it wrong, and Patterson would say, ‘Oh, you funny gorilla.’ ” More weirdly was this lawsuit against Patter…

Post by Infinity315 » Fri, Oct 31, 2025, 7:42 PM UTC

alyxya wrote: post #161

usrbinbash wrote: post #149

Earlier quoted context omitted.

> I don't think there's evidence that this issue would persist after continuing to scale models to be larger and doing more RL And how much larger do we need to make the models? 2x? 3x? 10x? 100x? How large do they need to get before scaling-up somehow solves everything? Because: 2x larger, means 2x more memory and compute required. Double the cost or half the capacity. Would people still pay for this tech if it doub…

Scaling applies to multiple dimensions simultaneously over time. A frontier model today could be replicated a year later with a model half the size, with a quarter of the FLOPS, etc. I don’t know the real numbers for optimization scaling, but you could check out NanoGPT speedrun [1] as an example. The best solution in the meantime is giving the LLM a harness that allows tool use like what coding agents have. I suspec…

Post by aoeusnth1 » Fri, Oct 31, 2025, 7:45 PM UTC

raddan wrote: post #29

equinox_nl wrote: post #3

But I also fail catastrophically once a reasoning problem exceeds modest complexity.

Yes, but you are not a computer. There is no point building another human. We have plenty of them.

Post by galaxyLogic » Fri, Oct 31, 2025, 7:49 PM UTC

ftalbot wrote: post #49

My_Name wrote: post #45

I find that they know what they know fairly well, but if you move beyond that, into what can be reasoned from what they know, they have a profound lack of ability to do that. They are good at repeating their training data, not thinking about it. The problem, I find, is that they then don't stop, or say they don't know (unless explicitly prompted to do so) they just make stuff up and express it with just as much confi…

Every token in a response has an element of randomness to it. This means they’re non-deterministic. Even if you set up something within their training data there is some chance that you could get a nonsense, opposite, and/or dangerous result. The chance of that may be low because of things being set up for it to review its result, but there is no way to make a non-deterministic answer fully bound to solving or reason…

Post by HarHarVeryFunny » Fri, Oct 31, 2025, 8:10 PM UTC

gpderetta wrote: post #151

dymk wrote: post #148

Earlier quoted context omitted.

This is too large of an oversimplification of how an LLM works. I hope the meme that they are just next token predictors dies out soon, before it becomes a permanent fixture of incorrect but often stated “common sense”. They’re not Markov chains.

Indeed, they are next token predictors, but this is a vacuous statement because the predictor can be arbitrary complex.