Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

quesma.com

Post reply on HN
Print view View original

61–70 of 137 posts

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#61

Post by lowbloodsugar »

SwellJoe wrote: post #14
spider-mario wrote: post #7

Earlier quoted context omitted.

> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it. Have you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.

Even at 30 t/s, 3.8 thinks so long, even on medium, it still takes 3x or more longer than any cloud model, in my testing.

I've got an M1 Max 64GB too. It's just not an LLM-class workstation. Give it a year and buy an M7 and you'll be laughing. Right now is a really bad time to invest in anything - using the cloud is the cheapest option, especially for open weight models.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#62

Post by selectodude »

purpleflame1257 wrote: post #2

There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.

I have a 5080, three OpenAI Pro token resets, and I’m on paternity leave. Astra seems pretty clever. Maybe I’ll give it a task.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#63

Post by anyfoo »

I have a very interesting self-made coding benchmark, very intricate and technical, but 100% a real world problem I had to solve. I’m not going to further elaborate, since I don’t want future models to train on the solution.

To my own surprise, Q6_K_XL (from unsloth) comes up with a solution, anything Q5 doesn’t. To further surprise me, so far only the XL Q6 variant managed to solve it.

The problem, at least as stated, seems to be right on the edge of what the Q6 quantization can do.

Unfortunately even a successful run is rather long, so I don’t have a whole lot of data.

But the whole thing sure made me doubt the common idea that you wouldn’t perceive a difference until crossing past 4 bits quantization.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#64

Post by sidewndr46 »

bellowsgulch wrote: post #5

Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it. Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work th…

I've ran some agentic stuff with Qwen3.8-27B-UD-Q4_K_M on my RTX 3090. It's fast enough to be usable in my opinion. But Qwen3.6-35B-A3B in the same quantization is much faster

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#65

Post by TomBombadildoze »

kennywinker wrote: post #57
dotinvictim wrote: post #27

local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable

It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).

Beats Opus 4.6 at what exactly? It certainly isn't code.

I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#66

Post by anyfoo »

sharmajai wrote: post #39

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level. It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the…

Not for me. As stated elsewhere, even Q5 (!) seems to be enough to kill the model’s ability to solve a particular problem in reasonable time. But that might just be right at the edge of what the model can do in the first place.

I have another personal benchmark problem (of a very different nature) that Qwen3.8-27B usually can’t solve at all, while Opus and GLM-5.3-Flash solve it consistently and very beautifully.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#67

Post by spider-mario »

Xeoncross wrote: post #9
spider-mario wrote: post #7

Earlier quoted context omitted.

> Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it. Have you tried it with MTPLX? I get around 30 tok/s with it, also on an M1 Max with 64GB.

Nice, which model quantization is this? Is it on huggingface?

MTPLX is this software: https://github.com/youssofal/MTPLX

I tried it with the author’s 4-bit quant of Qwen 3.8 27B: https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized... (but no need to download it manually; MTPLX will ask which one you want).

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#68

Post by teaearlgraycold »

anyfoo wrote: post #63

I have a very interesting self-made coding benchmark, very intricate and technical, but 100% a real world problem I had to solve. I’m not going to further elaborate, since I don’t want future models to train on the solution. To my own surprise, Q6_K_XL (from unsloth) comes up with a solution, anything Q5 doesn’t. To further surprise me, so far only the XL Q6 variant managed to solve it. The problem, at least as state…

I thought the wisdom is more so don’t bother going below 4bit and you won’t see a difference above 8bit.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#69

Post by diseasedyak »

spider-mario wrote: post #4

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…

Yah, prediction interval instead, right? (I'm still learning statistics)

Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#70

Post by anyfoo »

teaearlgraycold wrote: post #68
anyfoo wrote: post #63

I have a very interesting self-made coding benchmark, very intricate and technical, but 100% a real world problem I had to solve. I’m not going to further elaborate, since I don’t want future models to train on the solution. To my own surprise, Q6_K_XL (from unsloth) comes up with a solution, anything Q5 doesn’t. To further surprise me, so far only the XL Q6 variant managed to solve it. The problem, at least as state…

I thought the wisdom is more so don’t bother going below 4bit and you won’t see a difference above 8bit.

Depends on the actual audience, I guess. My stated “wisdom” comes in part from /r/LocalLLaMa, and my impression is that the tasks that users there give their models to try them out lean towards rather simplistic, on the reasoning side.

But there I literally did read “you don’t need anything better than 4 bpw” a bunch of times.

Post reply on HN