Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B is excellent, but it defaults to overthinking things

simonwillison.net

Post reply on HN
Print view View original

331–340 of 411 posts

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#331

Post by paulbjensen »

digidecode wrote: post #239
paulbjensen wrote: post #237

I just used it on a Apple M4 MacBook Pro with 48GB RAM with llama.cpp and Pi to help diagnose an infinite looping request in a React Server component on a Next.js application. After about 10+ hours of digging, it has apparently found a bug in the Next.js framework, with an example app that replicates the bug, and a fix for now to disable prefetch in the Link component. I had in my prompt asked it to discover the root…

10 hours at what tokens per sec?

I was using this model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Quantization is Q4_K_M (4-bit K-quants, medium)

From Pi, these are the up/down token counts:

Tokens: ↑147k ↓110k R26.5M

Prompt submitted at 18:52:54 BST on Sunday 16th August 2026, and finished at 04:05:27 BST on Monday 17th August 2026.

Last print out from the llama.cpp server logs:

881.42.815.480 I slot print_timing: id 2 | task 110233 | prompt eval time = 2742435.76 ms / 134911 tokens ( 20.33 ms per token, 49.19 tokens per second) 881.42.815.483 I slot print_timing: id 2 | task 110233 | eval time = 6312248.28 ms / 5804 tokens ( 1087.57 ms per token, 0.92 tokens per second) 881.42.815.483 I slot print_timing: id 2 | task 110233 | total time = 9054684.04 ms / 140715 tokens 881.42.815.484 I slot print_timing: id 2 | task 110233 | graphs reused = 114795 881.42.820.381 I slot release: id 2 | task 110233 | stop processing: n_tokens = 140714, truncated = 0

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#332

Post by throw1234567891 »

tiahura wrote: post #302
DanielHB wrote: post #222

Earlier quoted context omitted.

I am very much a beginner to local LLM stuff and I find it incredibly hard to figure out how to run models optimally with the correct settings for my hardware. The number of different variations of the same model and how each quant work is super confusing as well. When I tried to run llama.cpp directly I was getting max 9tk/s on qwen3.5-9B, then I tried LM Studio with the same model and got 77tk/s. I haven't figured…

If your package manager / configurator isn’t claude code or codex, you’re wasting time.

Your funny.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#333

Post by 0-_-0 »

nsingh2 wrote: post #64
jedbrooke wrote: post #50

I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought

One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning. [1] https://github.com/sapientinc/HRM-Text

Reasoning is already taking place in latent space, the CoT is just a sideeffect

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#334

Post by 0-_-0 »

zarzavat wrote: post #326
vardhanw wrote: post #271

Earlier quoted context omitted.

So, does "reasoning in latent space" mean that it does/will not use tokens (e.g. using CoT etc.)?

Models already reason in latent space, but they have to keep encoding-decoding their "thoughts" from/to human interpretable tokens, and notably those tokens are sampled from a distribution. The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into. It's as if the only way you could think was by writing down a word, erasing al…

The kv cache is not discretised, only the output tokens.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#335

Post by fweimer »

janalsncm wrote: post #159
fweimer wrote: post #135

Earlier quoted context omitted.

Centralized inference can easily increase batch size, leading to huge efficiency gains in the usual scenario where most users have just one or very few session. Using local resources efficiently requires some way to increase the batch size. I'm not sure if we are there yet.

I think the point is that if people are able to run inference on their laptops batch size efficiency won’t matter. And before that, businesses will be able to get decent results with dedicated inference hardware.

I think we'd need something different to make that a reality today. It's likely that decentralized compute eventually wins out here as well. It usually does. But without architectural changes, it will take a couple of years before the current single-conversation flow is truly usable on consumer-grade hardware.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#336

Post by naasking »

To what extent can the overthinking be reduced by a better system prompt? For instance, telling it not to overthink, embellish or snazz anything up, and just directly deliver the most what was requested without in the plainest form to build on in future iterations.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#337

Post by wren6991 »

zarzavat wrote: post #326
vardhanw wrote: post #271

Earlier quoted context omitted.

So, does "reasoning in latent space" mean that it does/will not use tokens (e.g. using CoT etc.)?

Models already reason in latent space, but they have to keep encoding-decoding their "thoughts" from/to human interpretable tokens, and notably those tokens are sampled from a distribution. The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into. It's as if the only way you could think was by writing down a word, erasing al…

> The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into.

Not completely true: KV is a projection of the activation at each layer's input, so attention heads see (a representation of) all previous tokens' activations at that layer. The hard decision at the LM head doesn't change that.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#338

Post by Anonyneko »

blagui wrote: post #22

You have 4 thinking levels. You can disable it. It's well known issue in Qwen, previous releases I would disable it by default. Also xhigh seem a new thing.

So it can be disabled on 27B after all? I haven't downloaded 3.8 27B yet, but the announcement made it seem like only the hosted Max can disable thinking, and with the open weights you could only select between provided reasoning efforts.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#339

Post by Bender »

chvid wrote: post #66

“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.” I think that should be the blinking headline - this shows what can be done with consumer hardware.

Do these self hosted models avoid "protecting the user" or protecting big businesses? In other words can I just ask it any question and if it has the answer, I will get an answer rather than telling me it can't answer the question?

I ask because Claude is fun for rewriting abandoned code and I am not a proper developer so it's been great for me. Claude refuses to answer questions about science and medicine that stray outside of the officially supported narratives of the AMA and I have issues that have surpassed anything a doctor can do so I am entirely on my own. Will the self hosted models answer such questions or will it also try to put walls or bumper guards around topics?

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#340

Post by pessimizer »

janalsncm wrote: post #159
fweimer wrote: post #135

Earlier quoted context omitted.

Centralized inference can easily increase batch size, leading to huge efficiency gains in the usual scenario where most users have just one or very few session. Using local resources efficiently requires some way to increase the batch size. I'm not sure if we are there yet.

I think the point is that if people are able to run inference on their laptops batch size efficiency won’t matter. And before that, businesses will be able to get decent results with dedicated inference hardware.

Also, an entire family can share one big box that can run a good model. Not an entire household, but an entire family no matter where they live in relation to the box. They could all access it over the internet, or even simply over the telephone. When it becomes another family member/servant, a price tag in the low $10s of thousands seems a lot more reasonable - and this is accessible now.

I could absolutely see local AI taking the place of voicemail/call screening completely. Call me and you get my AI, who will route the call to me if approved, or even choose to service the call itself. If a friend of mine calls who doesn't have $20K to spend on a great AI rig, they would certainly have permission to steal otherwise wasted cycles from mine. An answering machine isn't much different than an issue tracker, and some of those are 97% AIs having perfectly intelligible conversations with each other, and 3% humans being eagerly serviced by AI.

The old objection was "this is so hard to set up, nobody wants to run a server!" Now it can set itself up.

tldr; you don't have to rent from some data center. You can get high utilization out of a local box. This 1) puts a hard ceiling on what a data center can charge, and 2) they won't be able to compete on privacy, which locally can be complete.

Post reply on HN