Live data from Hacker News

Qwen 3.8 27B is excellent, but it defaults to overthinking things

simonwillison.net

171–180 of 411 posts

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#171
post #66

“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.” I think that should be the blinking headline - this shows what can be done with consumer hardware.

For me that moment was Gemma 4 12B QAT. You're not suddenly going to start throwing your hardest programming problems at Gemma 4 12B QAT, it is still 15B parameters less. It's more that, aside from pelican art which isn't what local models are for, I didn't see anything on Simon's post that it couldn't assist with or largely succeed at.

It can run 80-100t/s on a laptop, can understand images natively and do bounding boxes, read tiny text, understands audio natively as well and can transcribe or translate anything you say, can do accurate long context retrieval with pretty large context windows, tool calling, excellent reasoning and is very token efficient.

It's only 7GB including the mmproj or 8GB with MTP. The Qwen 3.8 27B model Simon was using is ~18GB with MTP+mmproj, rather than 17GB alone. The point is not really that you compare these models directly, but that Gemma 4 12B QAT was really a special moment in model releases deserving of a similar reaction relative to its size, but was mutilated by Google themselves, Unsloth and Llama.cpp.

The overall appreciation I think we're seeing this year in particular is that people are easily surprised when multiple things are improving simultaneously which produce seemingly exponential changes. It isn't just that models are getting smaller, or that reasoning is getting better, or that speculative decoding is becoming mainstream, or that models can understand audio and images better now, or that they can reliably call tools which expands their capabilities, or that context windows are getting larger, or that accurate retrieval is improved, or that.... and so on. It's all of them narrowing in at once that is starting to make local models incredible and truly useful for far more use cases on the existing hardware people already have.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#172
post #83

Earlier quoted context omitted.

It should just about be usable in 32GB.

RAM is never the issue, it's always the compute power

Quite the opposite, RAM is always the issue. More specifically, high bandwidth RAM.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#173
post #70
post #38

All current era models overthink as it's a product of their RL incentives (or distillation of models with them...) From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like: Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively. That is fantastic for SWE benchmark…

The recent 'Stolen Thoughts'[1] paper shows many excerpts of private reasoning for frontier models. For a complex maths problem, Sol reasoned in 367 tokens before working: We need solve. Need interpret no person sits next to two other people = among binary string length16 weight8, no occupied chair whose both neighbors occupied, equivalently ab 111 substring. Endpoints cannot have two neighbors anyway; only avoid 111…

I wonder if a human learning to mimic this thinking style work would improve their thinking ability?

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#174
post #83

Earlier quoted context omitted.

It should just about be usable in 32GB.

RAM is never the issue, it's always the compute power

what??? not true!

for inference the compute is the last thing we need more of.

memory bandwidth is the numebr one blocker, after that the inefficiencies that where introduced with MoE models (and all new large models are made that way)

Here is a quick read: https://news.ycombinator.com/item?id=49324600

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#175

Am I the only one who enjoys it when LLMs overthink everything? Opus 4.8 would spend like 10 minutes thinking and then go out there and do an excellent job. Only Fable 5 seems to be smart enough to just know everything it needs to immediately start working without any reasoning or verification. Opus 5 tries to be relentless like Fable, but it's not as smart as Fable and I have to constantly challenge and correct its…

Depends on how you work with it. Reading the meandering and repetitive thinking is disturbing and taxing, so we can’t do that. So thinking has to be (mostly) hidden and just becomes waiting time.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#176

I forked llama.cpp and added some crude mechanism to keep exactly this behavior under control - essentially guiding the reasoning process by injecting text strategically at specific thresholds. This was mainly put together to rein in Qwen3.6-27B, but I'd imagine 3.8 would react similarly. Fork can be found here - https://github.com/laurencehardman/llama-mindcontrol/tree/ma... Of course hacks like this are not perfect…

Is that similar to what ggerganov is talking about here?

https://x.com/ggerganov/status/2089214161884414147

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#177

Earlier quoted context omitted.

Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier. * Many businesses don't need frontier level intelligence anyway. * It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.

I mean, I might be missing something, but isn't part of the idea with cloud infra that you can scale down as well? Large orgs with significant demand might go out and buy local LLM hardware, but most businesses probably don't want to bother dropping $2k on a box with a beefy GPU and would rather just pay the lowest subscription tier so their employees can occasionally make queries. Plus, you know, the whole economies…

$2k? LOL. That's not even the GPU budget, these days.

The economies are currently out of whack because of underproduction of components and memory, so LLM providers have a few years of runway to entrench. Plus, the whole capex Vs opex thing that helped AWS will help here too, for sure.

At some point, though, things will change. More production will come online, and providers will have to end the current speculative subsidizing and jack up prices.

It's a bit like the dot-com era: the initial rush to land-grab web portals and e-commerce sites eventually died, once enough skills and infrastructure came online, and the bubble burst.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#178
post #66

“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.” I think that should be the blinking headline - this shows what can be done with consumer hardware.

I wish we could have better hardware and I think the tech is there for a few years already.

I've gone in (too many) details last night with the calcs: https://news.ycombinator.com/item?id=49324600

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#179
post #115

Earlier quoted context omitted.

Like the old proverb: "The marvel is not that the bear dances well, but that the bear dances at all."

Indeed. LLMs resemble human intelligence in more or less the same way that the output of the TI-99/4A speech synthesizer resembles a human voice.

Not if an LLM over chat can fool most people they're talking to a human (which it can), where the TI-99 speech synthesizer voice absolutely can not.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#180

I forked llama.cpp and added some crude mechanism to keep exactly this behavior under control - essentially guiding the reasoning process by injecting text strategically at specific thresholds. This was mainly put together to rein in Qwen3.6-27B, but I'd imagine 3.8 would react similarly. Fork can be found here - https://github.com/laurencehardman/llama-mindcontrol/tree/ma... Of course hacks like this are not perfect…

Is that similar to what ggerganov is talking about here? https://x.com/ggerganov/status/2089214161884414147

Yes - it would seem so :)

I did make a PR to the official llama-cpp repo some time back (about a month or so), but abandoned it as there seemed to be too much community concern that the mechanism would degrade model performance... Perhaps i'll polish it up and put some effort into benchmarking and revive the project in the near future.

Post reply on HN