Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

quesma.com

Post reply on HN
Print view View original

41–50 of 137 posts

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#41

Post by anon291 »

sharmajai wrote: post #39

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level. It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the…

I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#42

Post by lowbloodsugar »

sharmajai wrote: post #39

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level. It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the…

If it digs itself into a hole, try low or medium. In the rust coding benchmarks (on my machine) it did better on low and medium because xhigh never finished.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#43

Post by Forgeties79 »

civvv wrote: post #15
purpleflame1257 wrote: post #2

There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.

Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!

What are you offloading to ram (or even CPU)? I’m using a 9080 (not XT) and having trouble with context/token rates

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#44

Post by stared »

Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

I wrote this blog post myself, with AI for proofreading (typos and grammar, but not style). There were a few singular sentences for which I had a writer's block, but not much besides that.

So, if there are irrelevant remarks, these are mine. :)

Charts are vibe-coded - but it took quite a bit of hand-holding to get something decent. And the logarithmic scale for model size is my conscious choice (against Claude's initial ideas).

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#45

Post by bellowsgulch »

spider-mario wrote: post #7
bellowsgulch wrote: post #5

Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it. Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work th…

> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it. Have you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.

Thanks, man! I’ll go use that now that I know. llama-server the last time I used it for inference with this model wasn’t able to produce work fast enough to reach those numbers.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#47

Post by vardalab »

Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

Just ask your freaking agent to read it for you and extract the information. That's what it's all about. Why would I be reading these articles other than information?

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#48

Post by kennywinker »

anon291 wrote: post #41
sharmajai wrote: post #39

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level. It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the…

I personally think thinking is basically variable but rate precision. If you are in a 4bit mode but need 2x as many tokens you're just doing fp8 with hoops( of course 4bit multiply is faster)

Fair enough mental model, except my GPU can’t load the 8bit version and paging from disk makes it way more than 1/2 speed.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#49

Post by kennywinker »

dofm wrote: post #28
jadbox wrote: post #11

Earlier quoted context omitted.

Q3 XL and Q3 XS are the two I'm trying to decide on

You might want to test this new dynamic GGUF: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (I don’t know much about it, just saw a YouTube video about it last night)

Another one to try:

https://huggingface.co/Jackrong/Qwopus3.8-27B-Flash-GGUF

Runs the 3bit model faster than the 2bit one runs on my old-ass card. Can’t vouch for its intelligence yet, but i suspect whatever loss in smarts it takes is made up for by the extra resolution.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#50

Post by wgd »

kouteiheika wrote: post #33

Note that these quants are not quantized uniformly, so 4-bit isn't actually a "true" 4-bit here, so these observations won't necessarily hold up to other quants which might be done differently.

It looks like they tested Q4_K_M which should be just the standard K-quant without any imatrix calibration. The smaller ones are indeed dynamic though.
Post reply on HN