I tried it and it performed poorly on my private benchmark problems. The overthinking problem is real, it takes 5-10x the reasoning tokens than comparable models. It is a sign of inadequate training of the base model and it is using more reasoning tokens to compensate for that. I also noticed that it is likely to get into somewhat repetitive reasoning and forgetting about some user requirements, suggesting that it could be the side effects of using 3:1 linear attention vs full attention.
Qwen 3.8 27B is excellent, but it defaults to overthinking things
101–110 of 411 posts
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#102Complaining about overthinking in xhigh then pointing out output had bugs with thinking turned off seems like it’s missing the obvious compromise?
Agreed. Test driving the bad default is what they deserve (they brought this upon themselves as a benchmaxx attempt), but comparing it to running with reasoning completely disabled is also weird.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#103To me, the amazing thing is that we now have local models that rival the reasoning of high end models from about a year ago. I hope this trend continues.
Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier. * Many businesses don't need frontier level intelligence anyway. * It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.
The whole cloud story lies on two aspects:
- Hyperscaling “we are going to serve billions of people in our applications”, which is becoming increasing unlikely as regional tech companies become more dominant than than the global one (this one is as much about geopolitics as technology)
- Operations is hard, in which case non-frontier models should be increasingly capable. Devops for small-ish deployment is one of the few cases where it is hard to clam you need deep expertise and AI can’t do it. Previously, the claim is that you need people specialized in ops, which is expensive. Now…
My prediction is that not just cloud LLM, but cloud business general will have to change. Not yet in the next 5 years, but probably 8-20 years-ish
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#104I do agree that Qwen 3.8 27B is excellent but slow and very token inefficient. My benchmark places it near opus 4.6 and codex 5.3 performance. 3.6 27B couldn't even complete the benchmark. Please see below for details: https://gist.github.com/nharziro/aed0c364ce2f295a493494c6f1b...
Opus 4.6 performance with a local model that can be hosted on consumer hardware is an incredible result!!
I was genuinely surprised because it's quite a leap from where 3.6 was an as far as I understand this isn't a new model, it's the same model that's been post trained, so I don't quite understand what they did to improve it so substantially. The previous model couldnt get through the benchmark at all. Though it remains terribly inefficient and slow. The hardware will have to get substantially faster for these kinds of models to be daily drivers. I think I forgot to mention in the bench that I ran it on an m5 max mac book
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#105Earlier quoted context omitted.
Tried yesterday on my own laptop (a UltraCore 7 255H without dedicated GPU,with 32 GB RAM), it wasn't even starting thinking, even on a small context window (65k)
I think not much can run without a dedicated GPU
Till now I was using successfully Qwen 3.5 and Gemma 4 at a reasonable speed
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#106This could turn nasdaq red tmr
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#107Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#108[flagged]
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#109I find that a lot of the recent allegedly great open models are cranking their reasoning way further than I find reasonable for interactive use. I’m writing this while waiting for the new Deepseek V4 Flash to finish its task, which is taking way longer than the older version.
What gets reported is always the benchmark result, but rarely the real-world trade-off made to achieve it. That’s an obvious incentive for the labs, so I think Simon is correctly zeroing in on it. Please continue doing so for models that don’t go too far as much as this release.
Don’t get me wrong, I think it’s amazing what we can get out of smaller models with more reasoning, but we should be super aware how very much not-free it is.
This is a good opportunity to call out models that reason quickly: Meta’s Glimmer seems to be pretty token efficient so far, as do the GPT 5.6s.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#110[dead]