Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B is excellent, but it defaults to overthinking things

simonwillison.net

Post reply on HN
Print view View original

251–260 of 411 posts

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#251

Post by mordae »

jedbrooke wrote: post #50

I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought

It needs to argue with itself to extract most of the knowledge embedded in the weights into the context. Asking it to synthesize ideas directly in a single go is simply unreasonable. And MoE models need to walk multiple experts to extract all the knowledge on top of that. So you need to give them the reasoning trace to first spill all the associations into.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#252

Post by XCSme »

sgt wrote: post #248
XCSme wrote: post #220

Earlier quoted context omitted.

Thanks for the feedback! That might be my fault, I am not a designer, and I asked for most UI decisions. I tried to use colors to diferentiate models, the site is very data dense and it's hard to make everything readable. I've spent hundreds of hours building it, not sure if I would call it slop, but I just suck at design, lol Any suggestions on how to improve it?

How about a complete rewrite that is not vibe coded? I'm only half serious. You're likely to use an LLM to help develop it, but don't let it take you for a ride. LLM's really struggle with data dense and robust designs. Have a look at https://diskprices.com/ - this is great UX because it's extremely functional, yet it has nearly no design to speak of. On a scale of data dense designs, this one would be on the extreme…

I am still pretty happy with some UX decisions I took:

- the spotlight search, you can do ctrl+k and search "gpt vs claude vs google" and it automatically shows the most relevant comparison

- drag to reorder compared models

- filter leaderboard by multiple comma separated needles, so you can see for example all qwen and kimi models only

- the showcases page is quite cool

A lot of invisible UI/UX features are there, which I think work great, only the visual presentation could be, as you mentioned, a lot better.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#253

Post by harhargange »

I had been planning to buy a GPU for Blender to compliment my 5950x CPU. The moment I ran the Qwen3.6-27b on my CPU, I arrived at the conclusion about the GPU I want. I saved some money and ordered the 7900xt-20gb for around 600 USD (instead of 7900xtx for 900USD, Nvidia out of question due to prices). I just ran the Qwen3.8-27b and asked it to benchmark itself. Here's the output: =================================================

stdout: Benchmarking model: qwen3.8:27b

=== A) 32k ctx, short prompt === Context window set to : 32768 Input (prompt) tokens : 32 Prompt processing : 68.5 tok/s (0.47s) Output tokens : 80 OUTPUT SPEED : 42.45 tok/s (1.88s) Wall-clock total : 3.9s

=== B) 65k ctx, short prompt === Context window set to : 65536 Input (prompt) tokens : 32 Prompt processing : 48.7 tok/s (0.66s) Output tokens : 80 OUTPUT SPEED : 20.49 tok/s (3.90s) Wall-clock total : 17.5s

Done.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#254

Post by cyanydeez »

dofm wrote: post #21
cyanydeez wrote: post #19

Earlier quoted context omitted.

to the point though: most of that overthinking is useless if you have a proper redirect message. So setting arbitrary budget and getting it a good message will do the trick regardless of what type of thinking it's doing. The reason thinking seems to work is that it's just trying to find an optimum outside the local optimum, and the thinking trace helps find it. The only think I could think that'd be better than the -…

> The reason thinking seems to work is that it's just trying to find an optimum outside the local optimum, and the thinking trace helps find it. Yes, I think I finally have an intuitive sense for that. But surely on a longer prompt it is still better for the final response if the thinking has at least brushed past all of the prompt? One of the things I witnessed with xhigh is that while the thinking trace starts out…

Sure, but interrupting it isn't going to stop it's next output or thought from continuing the progress.

I guess if you're literally in a greenfield project, at the start, and it has nothing else to look at, then you're right; but everything I've been doing is taking existing stuff and improving it.

So my advice might have a bad edge case that only exists immediately. My setup is to tweak and improve on existing code bases so there's almost always something it can review, write to, read, etc that'll drive it to where I'm trying to go. So even in a greenfield project, it's start to do something rather than overthink everything. Those thought traces don't suddenly disappear.

There is the "muddled middle" of context that might get trapped in, but I don't think anything you can really do will stop that.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#255

Post by mordae »

neuroticnews25 wrote: post #134
jedbrooke wrote: post #50

I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought

I was pretty happy with Depseek Pro in Opencode util I discovered I can see the thinking trace by clicking on the "thinking..." communicate. All those seemingly unnecessary "but wait" messages are frustrating to read. But at least to some extent it's just model taking time thinking through the problem, and the trace produced doesn't have to be representative of what happens internally: https://arxiv.org/abs/2404.1575…

It is MoE. It needs to engage multiple experts when the problem is complex or unclear. So you naturally see more of those simply as a primitive it learns to use to page in more diverse set of weights. Remember that each token is just 6 experts out of 256. So it literally needs to tell its router that it needs a different set the next time.

And this memory control primitive leaks into the reasoning chain, because it has no other channel for it available and we do not know how to train any other channel.

On the flip side, it tends to converge quickly, roughly proportional to the actual difficulty / clarity of the task.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#256

Post by kzrdude »

agile-gift0262 wrote: post #228
CMay wrote: post #218

Earlier quoted context omitted.

Google provided incorrect settings and an imperfect template. Unsloth modified the template and then finetuned their own version of the model to optimize for some benchmarks as a means of validating quants. Google and Llama.cpp then adopt template changes by default, so anyone downloading the new model or even using the original model will now automatically be using it incorrectly. Llama.cpp also uses the same infere…

And what's the right way to use Gemma? Where can I find the correct template and settings if those aren't the ones provided by Google, Unsloth, and aren't built into llama.cpp? I discarded using Gemma 4 because it got into weird loops when tool calling

Some weeks ago a new official Gemma 4 release was posted that corrected some of the chat template problems. So the official release files on hugging face should be the way to go.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#257

Post by Havoc »

ipdashc wrote: post #114
refactor_master wrote: post #33

Earlier quoted context omitted.

Unlike cloud infra in general which offers things like automatic backups, regional redundancy, and effectively unlimited scalability, it seems like the value proposition of cloud LLM gets ever shakier. * Many businesses don't need frontier level intelligence anyway. * It's completely stateless. If your local LLM machine catches fire? Nothing was lost. Buy another.

I mean, I might be missing something, but isn't part of the idea with cloud infra that you can scale down as well? Large orgs with significant demand might go out and buy local LLM hardware, but most businesses probably don't want to bother dropping $2k on a box with a beefy GPU and would rather just pay the lowest subscription tier so their employees can occasionally make queries. Plus, you know, the whole economies…

> isn't part of the idea with cloud infra that you can scale down as well?

Yes though anyone with an aws bill knows that’s sales pitch lies

The bill only goes one direction and it ain’t down

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#258

Post by kzrdude »

madduci wrote: post #105
aphroz wrote: post #91

Earlier quoted context omitted.

I think not much can run without a dedicated GPU

Till now I was using successfully Qwen 3.5 and Gemma 4 at a reasonable speed

I think (maybe I missed something) that identical size and quant versions of Qwen 3.5 and 3.8 should run at the same speed. It’s the exact same architecture.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#259

Post by mordae »

teravor wrote: post #60

when you distill a thinking LLM past its capacity it will default to overthinking because during training that was the only way for a chance at a reward on many tasks. you can generally avoid this if you specialize it on a domain that is within its capacity.

I think that in this case there is also the problem of trying to transfer MoE-style reasoning into a dense model. I mean, MoE needs reasoning to walk multiple experts, but dense model already has all the weights. So when you push it hard to reproduce the MoE traces, you are effectively asking a small mouse to role-play as an anthill. Not great.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#260

Post by madduci »

petu wrote: post #216
madduci wrote: post #87

Earlier quoted context omitted.

Tried yesterday on my own laptop (a UltraCore 7 255H without dedicated GPU,with 32 GB RAM), it wasn't even starting thinking, even on a small context window (65k)

What was your prompt length? It's possible it was just processing it and it's likely not fast on your setup.

Really small (<100 tokens), I wanted to test its capabilities
Post reply on HN