Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B

huggingface.co

Post reply on HN
Print view View original

691–700 of 848 posts

Re: Qwen 3.8 27B

#691

Post by jodleif »

pulse7 wrote: post #680
CMay wrote: post #518

Credit where it's due. Qwen 3.8 27B is only the second local model after Gemma 4 that managed to correctly reason through one of my private benchmarks. It took 5x as many tokens to do it and 12m30s with MTP enabled, but it did do it. Gemma 4 reasoned through it more implicitly, while Qwen 3.8 reasoned more explicitly. Laguna and Muse Glimmer failed hard on it, though they're useful for other tasks. The VRAM usage see…

Can you please tell which Gemma 4 variant managed to correctly reason through your private benchmarks? Was is Gemma 4 31B? What quantizations and context lengths did you use for Gemma 4 and Qwen 3.8 27B? I am asking because I can't even load Gemma 4 31B on my GPU with any reasonable quantization (even with small context), while I can run Qwen 3.8 27B with large context and good quantization...

Then you might be missing SWA. Gemma models are extremely memory hungry without

Re: Qwen 3.8 27B

#692

Post by LoganDark »

minimaltom wrote: post #214
LoganDark wrote: post #30

Earlier quoted context omitted.

Unfortunately, that chip just doesn't really have the memory bandwidth to run this (or nearly any) model at acceptable speeds. I have the exact same chip (M4 Max 128GB) and I've been trying to optimize a completely purpose-built implementation with Fable and this is just not possible. Even if you could reach the full 576GB/s, it's just physically impossible to exceed these numbers with the model's architecture: 2 bpw…

What is bpw? Also whats your cutoff for 'acceptable' speed? I would have said 25tok/s.

Bits per weight.

I consider 'acceptable speed' to be around 150t/s. Why? Well, this is generally what it takes to keep me engaged with the output, rather than immediately switching to other tasks and checking back later. When I check back later, I have more catch-up to do at once, and I haven't been following the process. So I have to recall it, familiarize myself with the new progress, and sort of get back into focus with it, which is a lot of mental work (even if it happens quickly in real-time). I prefer not to have to do this because of how much work it is, so I prefer to watch the agent in real-time and try to follow its reasoning. That also lets me interrupt it quickly when I see it about to make a mistake, or see an important detail I left out.

Re: Qwen 3.8 27B

#693

Post by mirekrusin »

satvikpendem wrote: post #158
Casteil wrote: post #26

Earlier quoted context omitted.

I'm hoping too that they'll put out some MoE variants. Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model. Edit: Like its predecessors, 3.8 seems really inclined to overthinking, and on a 27b dense model that's kind of painful. I think I'm going to stick with gemma4:26b-a3b as my go-to because it runs about 4x as fast and tends to only need a fraction of the tokens in its 'thinking' stage to get th…

Reduce or turn down thinking: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

[flagged]

Re: Qwen 3.8 27B

#694

Post by fragmede »

suprjami wrote: post #672
fragmede wrote: post #624

Earlier quoted context omitted.

So do I, but we live in the future: https://chatgpt.com/share/6a7fc3d2-39f4-83e8-a7c6-825ddfb5e7...

You might find this article relevant: https://news.ycombinator.com/item?id=49151933

I've run the inference to get the answers I linked to. If someone else does the same thing, that involves extra energy. If I read your conversation instead of generating my own, then that's one less tree that has to be chopped down.

Until we go advanced geothermal or we crack fusion, energy is dirty. Read my inference or link me to yours so we don't boil the planet.

Re: Qwen 3.8 27B

#695

Post by glinkot »

kimsey0 wrote: post #571

If anyone else is running this on an RTX 5090, https://github.com/Neroued/ninfer as inference engine gets me ~138 tokens/second, roughly double what I get with a naive llama.cpp setup.

Yep, same, testing it now and it flies!

Re: Qwen 3.8 27B

#697

Post by Der_Einzige »

nullc wrote: post #662
Der_Einzige wrote: post #180

Earlier quoted context omitted.

Yes yes, oh god yes. They also spread FUD in the form of terrible recommended sampler settings. If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.

Diverging from the sampler used in RL training is not good for long multi-turn results-- it's a great way to knock models into reasoning loops that wouldn't otherwise.

Peer reviews NeurIPS caliber paper to prove that? Because I can show you one titled "Long context generation is a sampling problem"...

Re: Qwen 3.8 27B

#698

Post by hadlock »

satvikpendem wrote: post #161

As usual, the Jinja templates are messed up so use this [0] to reduce or turn off thinking, fix tool calling, keep a 100% KV cache hit rate, etc. [0] https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

Thanks this bumped my agent success rate from 67% to 92.5% (!!!)

Re: Qwen 3.8 27B

#699

Post by CMay »

pulse7 wrote: post #680
CMay wrote: post #518

Credit where it's due. Qwen 3.8 27B is only the second local model after Gemma 4 that managed to correctly reason through one of my private benchmarks. It took 5x as many tokens to do it and 12m30s with MTP enabled, but it did do it. Gemma 4 reasoned through it more implicitly, while Qwen 3.8 reasoned more explicitly. Laguna and Muse Glimmer failed hard on it, though they're useful for other tasks. The VRAM usage see…

Can you please tell which Gemma 4 variant managed to correctly reason through your private benchmarks? Was is Gemma 4 31B? What quantizations and context lengths did you use for Gemma 4 and Qwen 3.8 27B? I am asking because I can't even load Gemma 4 31B on my GPU with any reasonable quantization (even with small context), while I can run Qwen 3.8 27B with large context and good quantization...

Gemma 4 12B, Gemma 4 12B QAT, Gemma 4 31B, Gemma 4 31B QAT

Gemma 4 26BA4B would get close, but not quite and sometimes even get stuck in loops despite a repeat penalty.

Do not use any newer updated templates or Unsloth fixes. Use older official templates that released with the models on the huggingface repo. The template here worked: https://huggingface.co/google/gemma-4-12B-it/tree/657684fef0...

llama-server --model "model.gguf" -fa on -np 1 --jinja --ctx-size 262144 -b 768 -ub 768 --cache-type-k f16 --cache-type-v q4_0 --repeat-penalty 1.1 --chat-template-file "chat_template.jinja"

If you don't explicitly point to the template file, then llama.cpp will either use the template inside the model file or it will use its own template copy and your results may vary. Obviously some of the template fixes are useful to people, so it depends if you're having problems with tool calling or can't fix the tool calling in other ways for your scenario.

My experience with the QAT models was that quantizing v to q4_0 gave me better results than q8_0 or even f16. I think the Gemma QAT models may have been QAT trained to expect a q4_0 quantized v cache. If you're not using a QAT model, I would leave both at f16.

Another thing aside from using the QAT models and a Q4_0 v cache since you're having trouble fitting the models, is that you don't have to use the mmproj if you don't intend to use vision. If you need vision, but are hurting on VRAM, then you should be using --no-mmproj-offload. That will keep the mmproj loaded in system RAM instead of on your GPU. Loading images will be a little bit slower, but it can still be quite fast and you'll have more breathing room on your GPU. If you don't provide the mmproj file on the command line at all, then it won't load it anyway. If you're using some program like LM Studio, a simple thing you can do is move the mmproj and mtp files out of the directory for the model so LM studio can't find them and then it won't load them at all.

For Qwen 3.8 27B, doing any quantizing definitely hurt results a lot, so in my case I used: llama-server --model "Qwen3.8-27B-UD-Q4_K_XL.gguf" --spec-type draft-mtp --spec-draft-p-min 0.35 --spec-draft-n-max 2 -fa on -np 1 --jinja --ctx-size 65536 -b 768 -ub 768 --cache-type-k f16 --cache-type-v f16

Re: Qwen 3.8 27B

#700

Post by MrDrMcCoy »

bilekas wrote: post #678
MrDrMcCoy wrote: post #625

Earlier quoted context omitted.

> "I went to youtube, gmail, ycombinator, deliveroo, then I went to another site I randomly chose, because they're the best, duh" The path I described is all within HuggingFace. > And How am I supposed to know that arriving to huggingface as a new user? Enlighten me. Because I told you, knowing it was the best starting point for newbies. > Cool.. Why don't they share em because I genuinely cant find em, I'm dumb. You…

> The path I described is all within HuggingFace. It's just not. Where do I see the model download ? The fact we're here is a loss. It's not intuitive. Deal with it, or fix it.

I gave you a direct link to where you can get a download, and explained why I chose to link you where I did. I can't post screenshots here drawing you a map of how to use a website. I don't work for Huggingface or any AI company and can affect no changes to how intuitive any of it is, and think is easy enough already.

Huggingface, Unsloth, and llama.cpp all have documentation you can follow that will exceed anything I can tell you here. LMstudio, Lemonade, or Ollama might be even easier for you to use. Take my suggestions or don't.

Post reply on HN