Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B is excellent, but it defaults to overthinking things

simonwillison.net

Post reply on HN
Print view View original

61–70 of 411 posts

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#61

Post by SwellJoe »

syntaxing wrote: post #32
SwellJoe wrote: post #16

Earlier quoted context omitted.

Yes, default everything, no tuning, 8_K_XL Unsloth quantization on dual Radeon V620 GPUs (which aren't blazing, but faster than the Strix Halo).

What speed do you get on this setup? Im tempted to use the same GPU.

It varies widely based on a bunch of factors. With this specific model at 8-bit quantization and MTP, it starts out at about 25 t/s for basic chat, but for agentic tasks with long context it slows way down to something like 12-15. I don't see a big difference in token rates based on any config changes I have tried, or going to the smaller 6-bit quantization, so far, though I haven't spent a ton of time on experimenting.

If you already have one or more of them, then, yeah, you can use them for this model or any other at around this size or smaller, but I wouldn't recommend you buy them (or anything else, right now, everything is just too overpriced). You can run better models for less money at higher speeds. I bought mine before they got more expensive, but I wish I'd just bit the bullet and bought newer/faster cards before they got more overpriced. Or, the actual smart money, even back then was to just use cloud models and forget about self-hosting.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#63

Post by suprjami »

jedbrooke wrote: post #50

I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought

So-called "caveman" thinking attempts to address this.

The important part of "actually wait, I really need to XYZ" is just "XYZ".

The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#64

Post by nsingh2 »

jedbrooke wrote: post #50

I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought

One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning.

[1] https://github.com/sapientinc/HRM-Text

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#65

Post by anon373839 »

dofm wrote: post #36
simonw wrote: post #6

Earlier quoted context omitted.

Was that with the default xhigh reasoning setting? I suggest trying again with reasoning set to low or turned off entirely.

You now have me testing it with reasoning turned off, which I have never bothered much with on any other local models because it's rarely worth it. The result appears to be almost as good as Qwen 3.6 35B A3B on medium thinking mode. It second-guesses a little, it gives broader/more speculative answers, of course, and it missed the nuance of one of my prompts, but this gives me a lot more confidence that the Low reaso…

> Qwen 3.6 35B A3B on medium thinking mode

Qwen 3.6 doesn’t have configurable reasoning effort, does it?

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#66

Post by chvid »

“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.”

I think that should be the blinking headline - this shows what can be done with consumer hardware.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#67

Post by SV_BubbleTime »

refactor_master wrote: post #33
RachelF wrote: post #30

To me, the amazing thing is that we now have local models that rival the reasoning of high end models from about a year ago. I hope this trend continues.

Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier. * Many businesses don't need frontier level intelligence anyway. * It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.

[dead]

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#68

Post by kennywinker »

fermuch wrote: post #23
SwellJoe wrote: post #4

This is true, but I think it understates the problem. I did a task I've done with a bunch of small models lately ( https://github.com/swelljoe/flar/pull/17 ), and it did an excellent job, the best of any self-hostable model. But, it took eleven (11!) hours on my dual GPU setup. It really chewed on it, and spent a lot of time checking and re-checking. It is by far the slowest model I've used for the task. GPT 5.5 did…

xhigh tells it to overthink and re check everything. Low tells it to only do the minimum thinking necessary. I would suggest to give qwen medium which doesn't inject any thinking directives into it and also to give as much context as you can, ideally around 500k tokens or even 1M if you can. Big complex tasks like these make the model hit the compaction trigger a lot and they end up re thinking the same thing several…

Doesn’t it max out its context at like 256k?

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#69

Post by kennywinker »

LoganDark wrote: post #46
dofm wrote: post #29

Earlier quoted context omitted.

AFAIK that is initially only for the iPhone?

Apple is reportedly considering skipping the higher-end M6 chips altogether, which could potentially give enough time for higher-end M7 (in over a year) to use HBM

Considering, or being forced to consider due to the hardware crunch?

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#70

Post by dannyw »

jatora wrote: post #38

All current era models overthink as it's a product of their RL incentives (or distillation of models with them...) From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like: Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively. That is fantastic for SWE benchmark…

The recent 'Stolen Thoughts'[1] paper shows many excerpts of private reasoning for frontier models.

For a complex maths problem, Sol reasoned in 367 tokens before working:

  We need solve. Need interpret no person sits next to two other people = among binary string length16 weight8, no occupied chair whose both neighbors occupied, equivalently ab 111 substring. Endpoints cannot have two neighbors anyway; only avoid 111. Count binary strings length16 weight8 avoiding 111. Need N mod1000. Compute stepwise perhaps runs of 1 length max2. Count via runs.
  
  [... cut in half for HN readability ... ]

  Check interpretation potentially "no person sits next to two other people": no seated person's chair adjacent to two occupied chairs. Exactly no three consecutive chairs selected. yes.
  
  Need reason step by step final boxed 907. Explain runs. Ensure people each select chair distinct subset (given subset count). Let's present.
That doesn't look like an overthinker to me, and matches my experiences. There's plenty of papers and research on reducing thinking verbosity/length while keeping as much quality as possible.

I think one of the bigger problems is that verbose, `max`-style thinking does generally lead to higher benchmark scores. And model vendors are incentivised to for benchmarks (at least to some extent).

[1] https://stolen-thoughts.com/

Post reply on HN