91–100 of 411 posts
“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.” I think that should be the blinking headline - this shows what can be done with consumer hardware.
Tried yesterday on my own laptop (a UltraCore 7 255H without dedicated GPU,with 32 GB RAM), it wasn't even starting thinking, even on a small context window (65k)
All current era models overthink as it's a product of their RL incentives (or distillation of models with them...) From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like: Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively. That is fantastic for SWE benchmark…
For a model this small it’s also a way of trading supply-constrained VRAM for inference time, which for self hosting consumers (and probably Chinese companies subject to export controls) is mostly good. I can imagine a curve where for a given level of “intelligence” you either need model size, or inference time (“test time compute”), and can somewhat trade one for the other.
This is exactly what was shown on Luna/Terra/Sol tradeoffs - Luna requires much higher reasoning efforts to approach Terra/Sol on lower reasoning. Which is fine, of course, no complaints - but true.
On the other hand it maybe do too much i.e. I asked "how we could test it?" and instead of answering it just actually wrote tests. But it was the same with Qwen 3.6.
Complaining about overthinking in xhigh then pointing out output had bugs with thinking turned off seems like it’s missing the obvious compromise?
Earlier quoted context omitted.
You now have me testing it with reasoning turned off, which I have never bothered much with on any other local models because it's rarely worth it. The result appears to be almost as good as Qwen 3.6 35B A3B on medium thinking mode. It second-guesses a little, it gives broader/more speculative answers, of course, and it missed the nuance of one of my prompts, but this gives me a lot more confidence that the Low reaso…
> Qwen 3.6 35B A3B on medium thinking mode Qwen 3.6 doesn’t have configurable reasoning effort, does it?
I'm talking about budget — I mean limiting it to 2048 tokens.
… for one or other of the models I tested at the same time, in llama-server, there is a dropdown that offered options (unlimited, max, medium which was 2048)
(I've tested so many of these things now that they are beginning to blur.)
I thought that was llama-server with Qwen 35B, just checked and it's not. Nor is it Gemma 4 26B. Perhaps it was Ternary Bonsai which I tested again and deleted earlier.
Anyway I took to clipping Qwen 3.6 35B at 2048 tokens reasoning in LM Studio and elsewhere, and it did OK at that (because it often loops like mad on an ambiguous prompt if not curtailed).
FWIW I just rechecked outputs and I am a bit over-optimistic when I say 3.8 27B 's non-thinking output is that good. I spotted a couple of subtle errors in my tests that Low thinking didn't fail on. It is good, but it is not quite Qwen 3.6 35B thinking level.
Earlier quoted context omitted.
xhigh tells it to overthink and re check everything. Low tells it to only do the minimum thinking necessary. I would suggest to give qwen medium which doesn't inject any thinking directives into it and also to give as much context as you can, ideally around 500k tokens or even 1M if you can. Big complex tasks like these make the model hit the compaction trigger a lot and they end up re thinking the same thing several…
Doesn’t it max out its context at like 256k?
But, ~256k is all I can fit on my dual 32GB setup, and it gets slow as heck at 256k, anyway. I think just convincing it to chew less and do more is probably the right solution for Qwen 3.8 27B, though I guess we need some benchmarks to get the feel for how much intelligence you lose by choosing a lower reasoning level.