Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B is excellent, but it defaults to overthinking things

simonwillison.net

Post reply on HN
Print view View original

41–50 of 411 posts

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#41

Post by xlayn »

Gracana wrote: post #39
xlayn wrote: post #34

I have this branch of llama.cpp that among other things (like patching the template to not break the kv cache, and saving conversations to disk so you can resume quickly days after) also accept the reasoning effort flag here https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev... I did testing and the reasoning effort can be set per message, I was not aware of the option of none mentioned by @xscott, I tested…

What’s that about the template breaking the kv cache?

this is my understanding, the default template keeps the thinking part but only for the last message, so the harness has to play along with the template and strip and add to keep the conversation matching what's there on the llama.cpp cache, but if the harness sends the thinking in every turn, then you break what llama.cpp expects, the conversation doesn't match anymore what you have on cache and it reprocesses again the whole conversation

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#42

Post by Gracana »

bogzz wrote: post #9
andy99 wrote: post #2

The big problem with overthinking on a dense model is obviously the speed hit you take. Going from Qwen 35BA3B to 27B for me is about 7-8x slower (should be ~9x?). This makes me a lot less patient for useless thinking tokens. I’d want to compare this to the new Muse 30B model which is super terse and has a whole different way of thinking (no “Wait,”) and in my experiments was way more token efficient to the point tha…

I love reading Glimmer's "thoughts". Why use many word when few do trick?

I’ve noticed dsv4 do that as well, but inconsistently. I thought it was broken at first, but no, it’s just kind of shorthand that it does while thinking.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#44

Post by johnnyApplePRNG »

According to the paper "Stealing reasoning traces from proprietary llms" [0] all frontier models overthink.

Thinking is good.

You just don't see it in proprietary harnesses because it's literally cryptographically hidden from you.

[0] https://arxiv.org/pdf/2608.09867

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#45

Post by Gracana »

xlayn wrote: post #41
Gracana wrote: post #39

Earlier quoted context omitted.

What’s that about the template breaking the kv cache?

this is my understanding, the default template keeps the thinking part but only for the last message, so the harness has to play along with the template and strip and add to keep the conversation matching what's there on the llama.cpp cache, but if the harness sends the thinking in every turn, then you break what llama.cpp expects, the conversation doesn't match anymore what you have on cache and it reprocesses again…

Oh, interesting. I want to say there’s a “preserve thinking” option in mainline llama.cpp, but I’m seeing mixed results in my searches and my in-flight internet is not good enough to dig into it properly. Good to hear you have a solution though, because that is likely something I’ll run into as well.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#46

Post by LoganDark »

dofm wrote: post #29
LoganDark wrote: post #14

I hope Apple does end up moving to HBM. Unified memory has been a huge godsend, but the low memory bandwidth is just such a killer. Even/especially on M5, where the available compute is starting to starve incredibly badly on ML workloads.

AFAIK that is initially only for the iPhone?

Apple is reportedly considering skipping the higher-end M6 chips altogether, which could potentially give enough time for higher-end M7 (in over a year) to use HBM

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#48

Post by hedgehog »

Gracana wrote: post #45
xlayn wrote: post #41

Earlier quoted context omitted.

this is my understanding, the default template keeps the thinking part but only for the last message, so the harness has to play along with the template and strip and add to keep the conversation matching what's there on the llama.cpp cache, but if the harness sends the thinking in every turn, then you break what llama.cpp expects, the conversation doesn't match anymore what you have on cache and it reprocesses again…

Oh, interesting. I want to say there’s a “preserve thinking” option in mainline llama.cpp, but I’m seeing mixed results in my searches and my in-flight internet is not good enough to dig into it properly. Good to hear you have a solution though, because that is likely something I’ll run into as well.

Look for improved templates by "froggeric" on Hugging Face. I use Qwen 3.6 a fair amount using his template and it fixes some issues I saw with the upstream versions.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#49

Post by bitexploder »

dofm wrote: post #26
bitexploder wrote: post #10

Earlier quoted context omitted.

I had to fix this on 35B A3B -- I have a proxy that just shuts it down if it gets to 2K thinking tokens and injects something like "We have thought enough, let's begin working." and it almost always finishes the turn then. It rarely needs more than 2K thinking tokens and if it does there is always next turn. I would need to see what 27B is actually doing, but these smaller Qwen models seem prone to this.

Unfortunately in xhigh reasoning effort it will burn through 2K tokens before it has even finished its bullet point overview. It really is intense and obsessive. You might need ten times more! Your strategy would likely help in medium reasoning effort (because there it gets caught up in the very typical Qwen looping). Not seen looping in the “low” reasoning effort mode.

I have been using Muse Glimmer for a few days instead of A3B. It gets the job done quicker than A3B despite being several times slower.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#50

Post by jedbrooke »

I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought
Post reply on HN