Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

quesma.com

Post reply on HN
Print view View original

81–90 of 141 posts

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#81

Post by skolos »

alentred wrote: post #58

I would be very interested in a similar benchmark for *KV cache* quantizations. I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there. I also heard that KV cache quantization matters more with longer contexts.…

On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different.

I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams. Also have about 2Gb vram left for use of OS.

my coding agents regularly reach 200k context used without noticeable degradation.

P.S. I used setup from: https://github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#82

Post by skolos »

alentred wrote: post #58

I would be very interested in a similar benchmark for *KV cache* quantizations. I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there. I also heard that KV cache quantization matters more with longer contexts.…

There was a study specifically related to Qwen3.8 27B that showed that kv cache quantization has almost no impact on this model all the way to q4:

https://arxiv.org/html/2609.04098

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#83

Post by KennyBlanken »

It's strange that the author has completely ignored the 3 bit quants which allow someone with a 16GB GPU to have 100-120k and still get full performance. You can't run any of the 4-bit quants on a 16GB gpu with enough context to be useful for all but the most basic tasks.

General purpose agents can need up to 30k just to reply with "1+1=2" because their prompting is so overloaded. 60-70k is decently usable, still not great for anything complex. A long running task in a general purpose agent can easily hit 100k.

What the vast majority of people care about is performance around what desktop consumer GPUs can run. 8GB, 10, 12, and 16GB of VRAM. What do models that will run at full performance, do?

Also important to know is how Qwen3.8-27B stacks up against qwen3.6-35B-A3B, which due to being MoE, will run on a 16GB card with plenty of speed 90% of the time, at higher quant - so you get more parameters and better quant. But 3.8 is supposed to be "better", so...?

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#84

Post by brynx97 »

civvv wrote: post #15
purpleflame1257 wrote: post #2

There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.

Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!

Could you comment more on how you set this up? I have a mostly idle 9070XT I use for gaming, and I was considering using it with the newer local open models. Many thanks.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#85

Post by wiml »

dofm wrote: post #31
cogman10 wrote: post #24

Earlier quoted context omitted.

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that. The line is "Is this an interesting and accurate article that concisely makes it's case". LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very…

My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem. In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.

This is a complaint I hear others voice and one I have myself. I think it comes down to the text being poorly written. Sure, LLMs are good at a lot of the surface indicators of good writing: they have a wide vocabulary, they use grammatical sentences, they break up the text into paragraphs, sections, and lists. But they're terrible at organizing the text and marshalling a concept to get it across to a specific audience. The section and paragraph breaks are meaningless, the metaphors are unenlightening, the rhythm is exhaustingly uniform. Pre-LLM, you'd find this kind of text in marketing materials, corporate PR, and heavily padded sophomore essays.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#87

Post by ricardobeat »

stared wrote: post #55
spider-mario wrote: post #4

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative". It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a B…

The main problem here is that a model that wildly fluctuates with 60% - 100% - 80% results will have the same wilson score as one that repeatedly scores 80% - 80% - 80%. So the 'confidence interval' bar is meaningless.

I'm not that well versed in statistics, but a standard box plot is probably the best alternative

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#88

Post by wiml »

wiml wrote: post #85
dofm wrote: post #31

Earlier quoted context omitted.

My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem. In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.

This is a complaint I hear others voice and one I have myself. I think it comes down to the text being poorly written. Sure, LLMs are good at a lot of the surface indicators of good writing: they have a wide vocabulary, they use grammatical sentences, they break up the text into paragraphs, sections, and lists. But they're terrible at organizing the text and marshalling a concept to get it across to a specific audien…

And reading my own comment it has the same kinds of problems (lookit all those 3-example lists!). I wouldn't hire me for an editing job.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#89

Post by civvv »

brynx97 wrote: post #84
civvv wrote: post #15

Earlier quoted context omitted.

Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!

Could you comment more on how you set this up? I have a mostly idle 9070XT I use for gaming, and I was considering using it with the newer local open models. Many thanks.

I got Qwen3.8-27B running on my RX 9070 XT by setting up AMD’s ROCm 7.14/TheRock runtime for gfx1201 and adding the required HIP/LLVM paths to Windows. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably.

I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#90

Post by civvv »

Forgeties79 wrote: post #43
civvv wrote: post #15

Earlier quoted context omitted.

Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!

What are you offloading to ram (or even CPU)? I’m using a 9080 (not XT) and having trouble with context/token rates

I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.
Post reply on HN