Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B is excellent, but it defaults to overthinking things

simonwillison.net

Post reply on HN
Print view View original

351–360 of 411 posts

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#351

Post by aesthesia »

mordae wrote: post #255
neuroticnews25 wrote: post #134

Earlier quoted context omitted.

I was pretty happy with Depseek Pro in Opencode util I discovered I can see the thinking trace by clicking on the "thinking..." communicate. All those seemingly unnecessary "but wait" messages are frustrating to read. But at least to some extent it's just model taking time thinking through the problem, and the trace produced doesn't have to be representative of what happens internally: https://arxiv.org/abs/2404.1575…

It is MoE. It needs to engage multiple experts when the problem is complex or unclear. So you naturally see more of those simply as a primitive it learns to use to page in more diverse set of weights. Remember that each token is just 6 experts out of 256. So it literally needs to tell its router that it needs a different set the next time. And this memory control primitive leaks into the reasoning chain, because it h…

Has this hypothesis that reasoning tokens help the model engage a wider range of experts been tested? I'm a little skeptical that it's the main driver of extended reasoning traces, particularly because MoE models are generally already trained so that expert activations are as uniformly distributed as possible. But it would be interesting to know to what extent this happens.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#353

Post by CamperBob2 »

zahlman wrote: post #210
coldtea wrote: post #179

Earlier quoted context omitted.

Not if an LLM over chat can fool most people they're talking to a human (which it can), where the TI-99 speech synthesizer voice absolutely can not.

> Not if an LLM over chat can fool most people they're talking to a human (which it can) I keep hearing this claim, and yet I keep seeing LLM output which is trivially distinguished from human writing. I really can't understand how this gap persists; but then, there seem to have been at least some people who couldn't sniff out ELIZA, back in the day, too.

I keep hearing this claim, and yet I keep seeing LLM output which is trivially distinguished from human writing.

That's because they're trained that way. If you trained a modern frontier LLM with the explicit goal of passing the Turing test, it would have no difficulty doing so.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#355

Post by dstryr »

romeinaday wrote: post #190

Can you run this on a 36GB MacBook Pro (M3 Pro)? What would be a good setup? for coding mainly

i have this exact same machine and 3.8-27b runs well, albeit slow at about 6 t/s.

i have it complete tasks in the background as i work on other things.

for comparison, my 3090 runs it at about 60 t/s.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#356

Post by dragonwriter »

Bender wrote: post #339
chvid wrote: post #66

“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.” I think that should be the blinking headline - this shows what can be done with consumer hardware.

Do these self hosted models avoid "protecting the user" or protecting big businesses? In other words can I just ask it any question and if it has the answer, I will get an answer rather than telling me it can't answer the question? I ask because Claude is fun for rewriting abandoned code and I am not a proper developer so it's been great for me. Claude refuses to answer questions about science and medicine that stray…

> Do these self hosted models avoid "protecting the user" or protecting big businesses? In other words can I just ask it any question and if it has the answer, I will get an answer rather than telling me it can't answer the question?

For the most part, open base models are corporate releases with somewhat similar guardrails to commercial hosted models (not quite as complete, because hosted models tend to have a combination of trained and external guardrails applied); but no one is monitoring and trying to terminate your account for using jailbreak prompts, and there are often community finetunes available that (among other things) weaken the trained-in guardrails.

Of course, even if the model does answer, it may nto answer according to the particular worldview that produces hostility to the “narratives of the AMA”.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#357

Post by shifto »

geye1234 wrote: post #325
shifto wrote: post #290

Local Qwen3.8 27B pointed to my local Silverbullet wiki coupled with a harness makes it so smart in my environment. It knows my network and homelab better than me. It can find things in an instant, troubleshoot problems on its own. Magic.

How do you deal with security issues? Up until now, I've only run agents on a separate machine on its own physical subnet. (Admittedly this is paranoid and I could probably run it on a VM.) I would love to give a local-LLM agent full access to my (very modest) homelab, but can't trust it not to delete my files or do something crazy. Maybe giving it its own non-root username, and therefore read access, allowing diagno…

I worry about this myself but find some solace and peace in my backup strategy. I treat the harness as a user and set permissive authorizations where needed. I even let it suggest why and which permissions it would need for specific items and note these down in the wiki as well. The idea is that the LLMs takes into consideration why it has the permissions it has and doesn't try and work around them.

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#358

Post by rao-v »

CMay wrote: post #218
podocarp wrote: post #193

Earlier quoted context omitted.

Out of the loop here. What did Google and unsloth and llama do to mutilate Gemma? I can understand Google shenanigans but llama and gunsmith is kind of surprising.

Google provided incorrect settings and an imperfect template. Unsloth modified the template and then finetuned their own version of the model to optimize for some benchmarks as a means of validating quants. Google and Llama.cpp then adopt template changes by default, so anyone downloading the new model or even using the original model will now automatically be using it incorrectly. Llama.cpp also uses the same infere…

I run llama.cpp and specialized forks on 64GB of HBM and I still cannot figure out where to find the final correct guidance on using the Gemma 4 models.

Would appreciate any kind of pointer to the latest!

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#359

Post by rene-veerman »

mattmaroon wrote: post #317
rene-veerman wrote: post #170

Earlier quoted context omitted.

how'd you get it to run? not even upgrading ollama could get it installed on my end :(

Oh I’m just using ChatGPT, I haven’t dives into getting anything to run locally yet. That’s on the off season project list.

i see. i'm eager to use the self-run versions of real code-assisting AIs you see. ah well, patience. more patience. .

Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things

#360

Post by rao-v »

dofm wrote: post #308
tarruda wrote: post #227

Earlier quoted context omitted.

> It can run 80-100t/s on a laptop That is a lot, what is your laptop hardware? One issue I have with Gemma is that they seem to use old architectures that rely on full attention, requiring a lot of RAM for context and quickly degrading speeds as context is filled. Qwen 3.5+ is much better in that regard with its super efficient context. Even on Macs, speeds take degrade much more slowly.

> One issue I have with Gemma is that they seem to use old architectures that rely on full attention, requiring a lot of RAM for context and quickly degrading speeds as context is filled. Yes, this is something I hope they will change. Gemma 4 31B is much slower on pre-Blackwell GPUs as a result, which is a bit of a shame for local model experimentation.

Even Muse Glimmer (as did GPT-OSS I think) does ~4 sliding window attention layers + 1 full attention layer (like Gemma 4). I’m assuming both labs have good reason to think that gated delta nets are not optimal.

Of course it’s possible the labs just stick with the optimal architecture for large models and GDN is best for smaller models.

Post reply on HN