Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

quesma.com

Post reply on HN
Print view View original

111–120 of 138 posts

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#111

Post by bitwize »

qsbuilder wrote: post #109

I always wonder is it safe to run one of these models on a personal pc, or do you guys recommend something like docker, sorry a bit new to all of this.

Yes, it's fine to run a model on bare metal. The model is just a token predictor. Leave out the fine semantics about this; it's a function taking a set of input tokens to output tokens.

It can't mess with your computer or files until you hook it to a harness, which interprets some of the model's output as commands to execute. So, model on bare metal, harness in a container or VM.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#112

Post by skirmish »

slim wrote: post #106
7speter wrote: post #91

Earlier quoted context omitted.

You can offload the vision projector to CPU/sysRAM

Awesome. How would you setup that ?

  --no-mmproj-offload
See the documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/multi...

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#114

Post by redox99 »

skolos wrote: post #81
alentred wrote: post #58

I would be very interested in a similar benchmark for *KV cache* quantizations. I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there. I also heard that KV cache quantization matters more with longer contexts.…

On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different. I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams. Also have about 2Gb vram left for use…

Those are really nice numbers. With that t/s, no network latency or queueing it must feel much snappier than cloud models.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#115

Post by noir_lord »

Winfred-zz wrote: post #103
dotinvictim wrote: post #27

local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable

In my own experience, qwen3.8-27b 4bit can consistently find bugs in software written by sonnet 5 and opus 5. But it does do that at maybe 1/10th the speed. Still a pretty good deal if you're coding without wanting to spend big. qwen3.8-27b 4bit has a following specifically for being exceptionally gifted for such a small model.

It’s genuinely the first local model that has actually made me keep it around, it’s pretty good at spotting things existing tooling can’t, like comment rot/drift/summarising accurate git changes, it’s also the first local model that pushes back when it thinks it is right (I don’t dislike that even when it is wrong, it still hallucinates things that don’t actually work the way it confidently states they do).

It does clear the useful enough to be worth it bar though.

Zero interest in remote models but local ones if they offer utility, sure.

Runs pretty well on a 7900XTX as well.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#116

Post by mitxela »

spider-mario wrote: post #4

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…

> The key confusion underlying the FCF is the confusion of what is known before observing the data — that the CI, whatever it will be, has a fixed chance of containing the true value — with what is known after observing the data. Frequentist CI theory says nothing at all about the probability that a particular, observed confidence interval contains the true value; it is either 0 (if the interval does not contain the parameter) or 1 (if the interval does contain the true value).

They're really reaching here. There's a 95% chance it's 1 and 5% it's 0, and we don't know which, so it's 0.95. Like saying the probability your uncle died of a shark attack is either 0 or 1. But in reality I don't know anything about your uncle so it's the average probability that someone picked randomly died of a shark attack, until I know it about your uncle.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#117

Post by zenoprax »

civvv wrote: post #15
purpleflame1257 wrote: post #2

There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.

Running Q3 on my AMD RX 9070XT. 32k context and 32/TPS. Apart from the context window preventing it from doing any large tasks, this thing is seriously powerful. I could probably push it to 64k context. Local open models are the future, and I am definitely getting a more powerful card. Very fun!

9070XT operator here: I'm using llama.cpp with the same model and quant and I'm getting 87,000 for my context limit. I tried the Unsloth models but they lowered it to around 30-40K so I went back to upstream.

I'm on Linux and using some sort of unholy mess of ROCM libraries that I don't understand.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#119

Post by skolos »

redox99 wrote: post #114
skolos wrote: post #81

Earlier quoted context omitted.

On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different. I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams. Also have about 2Gb vram left for use…

Those are really nice numbers. With that t/s, no network latency or queueing it must feel much snappier than cloud models.

it is, unless it set to xhigh - it really likes generating tons of tokens for its thinking. unfortunately, for decently reliable coding results you want it on xhigh ...
Post reply on HN