Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

quesma.com

Post reply on HN
Print view View original

101–110 of 137 posts

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#101

Post by kmike84 »

Measuring quality e2e definitely makes sense.

But I think there is a bit more to this:

> Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks.

A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbers more comparable? It's not measuring how well the model follows bf16 on agentic tasks.

I've been trying to check KLD recently for some quants of Qwen 3.8 27B, and the numbers are dramatically different, depending on which dataset you use. KLD computed on agentic traces is much higher, and top-1 % is way lower than if you compute it on chats or wiki text.

You look at a published number, and see "oh, nice, top1 is 99% - quant is different just in 1 token out of 100", but chances are it's computed on wiki, and on agentic / coding it can be 10 tokens out of 100.

Common intuition is that on agentic tasks errors compound, and that's why it degrades more than metrics show - but maybe the metrics themselves are also wrong, too optimistic.

Still investigating it though :)

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#102

Post by kennywinker »

fshr wrote: post #99
kennywinker wrote: post #57

Earlier quoted context omitted.

It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).

> 5060 I think you're referring to a 5060Ti 16GB, yes? 32k context is easily done there. 64k can work with a more aggressive quant, but you lose a bit of speed.

Yeah 16gb. For longer context, a 3bit quant is needed. Yes it’s tight on a 16gb card - can’t wait for the bubble to pop so hardware prices fall.

But I don’t quite follow you - how does a more aggressive quant slow it down? Less bits per token means faster inference not slower.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#103

Post by Winfred-zz »

dotinvictim wrote: post #27

local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable

In my own experience, qwen3.8-27b 4bit can consistently find bugs in software written by sonnet 5 and opus 5. But it does do that at maybe 1/10th the speed. Still a pretty good deal if you're coding without wanting to spend big.

qwen3.8-27b 4bit has a following specifically for being exceptionally gifted for such a small model.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#104

Post by dofm »

jameshart wrote: post #100
wiml wrote: post #85

Earlier quoted context omitted.

This is a complaint I hear others voice and one I have myself. I think it comes down to the text being poorly written. Sure, LLMs are good at a lot of the surface indicators of good writing: they have a wide vocabulary, they use grammatical sentences, they break up the text into paragraphs, sections, and lists. But they're terrible at organizing the text and marshalling a concept to get it across to a specific audien…

Possibly an inevitability, since LLM text is specifically generated to be low entropy. It’s smooth, optimized for minimal perplexity. And if you don’t find any of the word choices a little bit perplexing, your brain can just gloss over it. There is no meaning here. It’s like smalltalk.

Interesting — do you happen to have any suggestions for something I could read about this?

It really bugs me, this aspect of LLMs, and yet I find myself sounding like a lunatic when I describe it, until others chime in and say they have the same experience.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#105

Post by D13Fd »

skolos wrote: post #81
alentred wrote: post #58

I would be very interested in a similar benchmark for *KV cache* quantizations. I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there. I also heard that KV cache quantization matters more with longer contexts.…

On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different. I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams. Also have about 2Gb vram left for use…

I’m running 27B on a 5090 as well, and the results have been really strong. It does almost as well as, and sometimes better than, a 121gb DS4 model running on an M5 Max 128gb. 27B also flies on the 5090, and at medium think it returns results many times faster than my DS4 setup (the default xhigh is basically broken, though).

For the kinds of things I use a local model for (legal document review), it’s just spectacular. It also has good vision support. I’ve been using 27B more and more over DS4.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#106

Post by slim »

7speter wrote: post #91
civvv wrote: post #90

Earlier quoted context omitted.

I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.

You can offload the vision projector to CPU/sysRAM

Awesome. How would you setup that ?

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#107

Post by breezybottom »

stared wrote: post #55
spider-mario wrote: post #4

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative". It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a B…

Both ways involve sophistry. If you don't like dirty tricks, statistics isn't for you.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#108

Post by breezybottom »

cogman10 wrote: post #24
Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that. The line is "Is this an interesting and accurate article that concisely makes it's case". LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very…

It's not nearly as annoying as being tricked into reading them.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#110

Post by jmalicki »

spider-mario wrote: post #78
diseasedyak wrote: post #69

Earlier quoted context omitted.

Yah, prediction interval instead, right? (I'm still learning statistics) Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.

Yes. It’s maybe easier to reason about by imagining that we are trying to estimate the parameter of a Gaussian distribution. Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that. 95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.…

> a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well

Correct, which is why it's crazy to consider a Jeffrey's credible interval as valid, but not a confidence interval.

Perhaps the confidence interval might not be valid under a frequentist interpretation, but the same math can be valid under a Bayesian interpretation even if the person publishing didn't mean it that way.

Does that mean it's valid? Well, the confidence interval is as equally valid as a lot of credible intervals, which is why the credible interval is not some magical better alternative - it's mostly just a choice of how you interpret it, it is not a different procedure that produces different numbers.

Post reply on HN