Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B is excellent, but it defaults to overthinking things

simonwillison.net

Post reply on HN
Print view View original

321–330 of 411 posts

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#321

Post by CMay »

dofm wrote: post #305
CMay wrote: post #171

Earlier quoted context omitted.

For me that moment was Gemma 4 12B QAT. You're not suddenly going to start throwing your hardest programming problems at Gemma 4 12B QAT, it is still 15B parameters less. It's more that, aside from pelican art which isn't what local models are for, I didn't see anything on Simon's post that it couldn't assist with or largely succeed at. It can run 80-100t/s on a laptop, can understand images natively and do bounding…

> It's only 7GB including the mmproj or 8GB with MTP. Even more impressively it doesn't have a separate mmproj at all — it is fully integrated, and the vision encoder doesn't speak words into the LLM, as it were —- it is directly integrated into the model's weights. I have banged on about this model here enough but I really agree that Gemma 4 12B is a candidate for the most impressive LLM of the year. It is remarkabl…

Well in my case I'm using llama.cpp and the mmproj is required, but I think it is just an extracted part of the original model file. Even with audio, yes it technically supports them natively and they're "encoder-free", but in practice that doesn't mean no translation or processing is required before it goes into the model. It does require much less processing though, which reduces latency.

As for coding, for sure there are many important details that a model needs to know in order to produce correctness and the smaller a model is the more it ends up training out. If there's a task you do consistently enough though, often times you can simply provide a pile of essential context so it has good enough reference to not need the extra training data.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#322

Post by rpbiwer2 »

CMay wrote: post #283
kzrdude wrote: post #256

Earlier quoted context omitted.

Some weeks ago a new official Gemma 4 release was posted that corrected some of the chat template problems. So the official release files on hugging face should be the way to go.

The updated version will handle tool calling better by default, but the reasoning quality is no longer preserved and is mutilated quite badly.

So then how do you run it unmutilated?

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#323

Post by Sharlin »

vardhanw wrote: post #271
entrope wrote: post #219

Earlier quoted context omitted.

"Latent" means it (the space) is there all the time but usually not exposed or apparent. Latent space is a high-dimensional space that is not legible to humans, so normally reasoning gets constrained to progress in tokens instead so that it can be evaluated. I think https://arxiv.org/abs/2412.06769 is a good description of the premise of reasoning in latent space, although https://arxiv.org/abs/2604.15726 argues it's…

So, does "reasoning in latent space" mean that it does/will not use tokens (e.g. using CoT etc.)?

The computation still has to be done, even if it's more efficient in a higher-entropy space.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#324

Post by blactuary »

tiahura wrote: post #302
DanielHB wrote: post #222

Earlier quoted context omitted.

I am very much a beginner to local LLM stuff and I find it incredibly hard to figure out how to run models optimally with the correct settings for my hardware. The number of different variations of the same model and how each quant work is super confusing as well. When I tried to run llama.cpp directly I was getting max 9tk/s on qwen3.5-9B, then I tried LM Studio with the same model and got 77tk/s. I haven't figured…

If your package manager / configurator isn’t claude code or codex, you’re wasting time.

Some of us prefer to avoid Anthropic/OpenAI

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#325

Post by geye1234 »

shifto wrote: post #290

Local Qwen3.8 27B pointed to my local Silverbullet wiki coupled with a harness makes it so smart in my environment. It knows my network and homelab better than me. It can find things in an instant, troubleshoot problems on its own. Magic.

How do you deal with security issues? Up until now, I've only run agents on a separate machine on its own physical subnet. (Admittedly this is paranoid and I could probably run it on a VM.)

I would love to give a local-LLM agent full access to my (very modest) homelab, but can't trust it not to delete my files or do something crazy. Maybe giving it its own non-root username, and therefore read access, allowing diagnosis but not fixing. But even a non-root user can do some damage, or exfiltrate system info just by using curl.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#326

Post by zarzavat »

vardhanw wrote: post #271
entrope wrote: post #219

Earlier quoted context omitted.

"Latent" means it (the space) is there all the time but usually not exposed or apparent. Latent space is a high-dimensional space that is not legible to humans, so normally reasoning gets constrained to progress in tokens instead so that it can be evaluated. I think https://arxiv.org/abs/2412.06769 is a good description of the premise of reasoning in latent space, although https://arxiv.org/abs/2604.15726 argues it's…

So, does "reasoning in latent space" mean that it does/will not use tokens (e.g. using CoT etc.)?

Models already reason in latent space, but they have to keep encoding-decoding their "thoughts" from/to human interpretable tokens, and notably those tokens are sampled from a distribution. The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into.

It's as if the only way you could think was by writing down a word, erasing all the thoughts from your head, then reading the word you just wrote down and deciding on the next word, etc.

Reasoning purely in latent space means that the model would still produce an output equivalent to tokens but unconstrained e.g. the output could be raw and opaque vectors. A significant downside is that you lose the ability to inspect the reasoning trace. It would also make the reasoning trace potentially larger which has operational issues.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#327

Post by jacquesm »

zarzavat wrote: post #326
vardhanw wrote: post #271

Earlier quoted context omitted.

So, does "reasoning in latent space" mean that it does/will not use tokens (e.g. using CoT etc.)?

Models already reason in latent space, but they have to keep encoding-decoding their "thoughts" from/to human interpretable tokens, and notably those tokens are sampled from a distribution. The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into. It's as if the only way you could think was by writing down a word, erasing al…

More like erasing all other branches just prior to that word from your head, the past thoughts would still remain.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#328

Post by hedgehog »

rpbiwer2 wrote: post #322
CMay wrote: post #283

Earlier quoted context omitted.

The updated version will handle tool calling better by default, but the reasoning quality is no longer preserved and is mutilated quite badly.

So then how do you run it unmutilated?

Log all the calls and run on a periodic cadence (cron or ever N turns) a larger model (like Opus) to read samples of the traces and edit the template to fix observed problems. There are some signs that help find interesting things to look at, errors of course, but also overly long responses, prefix cache misses, tool call errors, etc.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#329

Post by stellamariesays »

chvid wrote: post #66

“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.” I think that should be the blinking headline - this shows what can be done with consumer hardware.

[flagged]

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#330

Post by raducu »

coldtea wrote: post #275
shafyy wrote: post #200

Earlier quoted context omitted.

Can it? I feel like I instantly recognize if I am chatting with an LLM or a human

"Feel" is doing a lot of work here. You can recognize 70% of those (true positive rate) and still have a false negative rate of 30%, while thinking you got 100% of the AI ones! The problem is that you'd be oblivious to those you don't recognize.

But the Turing test involves a conversation you can steer and I think that's where indeed it's a lot easier to smell LLMs.
Post reply on HN