Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

quesma.com

Post reply on HN
Print view View original

91–100 of 141 posts

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#91

Post by 7speter »

civvv wrote: post #90
Forgeties79 wrote: post #43

Earlier quoted context omitted.

What are you offloading to ram (or even CPU)? I’m using a 9080 (not XT) and having trouble with context/token rates

I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.

You can offload the vision projector to CPU/sysRAM

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#92

Post by [deleted] »

wgd wrote: post #50
kouteiheika wrote: post #33

Note that these quants are not quantized uniformly, so 4-bit isn't actually a "true" 4-bit here, so these observations won't necessarily hold up to other quants which might be done differently.

It looks like they tested Q4_K_M which should be just the standard K-quant without any imatrix calibration. The smaller ones are indeed dynamic though.

[deleted]

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#93

Post by stared »

ricardobeat wrote: post #87
stared wrote: post #55

Earlier quoted context omitted.

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative". It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a B…

The main problem here is that a model that wildly fluctuates with 60% - 100% - 80% results will have the same wilson score as one that repeatedly scores 80% - 80% - 80%. So the 'confidence interval' bar is meaningless. I'm not that well versed in statistics, but a standard box plot is probably the best alternative

A single result is binary. All we get from a run is which tasks were solved, which weren’t.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#95

Post by dofm »

wiml wrote: post #85
dofm wrote: post #31

Earlier quoted context omitted.

My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem. In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.

This is a complaint I hear others voice and one I have myself. I think it comes down to the text being poorly written. Sure, LLMs are good at a lot of the surface indicators of good writing: they have a wide vocabulary, they use grammatical sentences, they break up the text into paragraphs, sections, and lists. But they're terrible at organizing the text and marshalling a concept to get it across to a specific audien…

The weird, sort of sub-surface uniformity is what I notice. It appears to be doing all the right things, but you can't feel the writer.

It is like a rollercoaster that doesn't have any jerk or jounce.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#96

Post by anyfoo »

lowbloodsugar wrote: post #42
sharmajai wrote: post #39

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level. It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the…

If it digs itself into a hole, try low or medium. In the rust coding benchmarks (on my machine) it did better on low and medium because xhigh never finished.

I observed this as well. I have problems that get solved very quickly using non-thinking mode, but in thinking mode it digs itself into giant holes, that it sometimes doesn't manage to escape from.

Like wanting to implement an entire emulator to figure out something it could easily glean otherwise (and does, in non-thinking mode).

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#97

Post by brandon272 »

TomBombadildoze wrote: post #65
kennywinker wrote: post #57

Earlier quoted context omitted.

It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).

Beats Opus 4.6 at what exactly? It certainly isn't code. I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.

What configuration? What harness? These matter greatly to how a local model performs, in my experience.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#98

Post by tomrod »

stared wrote: post #55
spider-mario wrote: post #4

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative". It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a B…

I'm working on a practical review implementation on this! Great to hear others are thinking along the same way.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#99

Post by fshr »

kennywinker wrote: post #57
dotinvictim wrote: post #27

local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable

It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).

> 5060

I think you're referring to a 5060Ti 16GB, yes?

32k context is easily done there. 64k can work with a more aggressive quant, but you lose a bit of speed.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#100

Post by jameshart »

wiml wrote: post #85
dofm wrote: post #31

Earlier quoted context omitted.

My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem. In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.

This is a complaint I hear others voice and one I have myself. I think it comes down to the text being poorly written. Sure, LLMs are good at a lot of the surface indicators of good writing: they have a wide vocabulary, they use grammatical sentences, they break up the text into paragraphs, sections, and lists. But they're terrible at organizing the text and marshalling a concept to get it across to a specific audien…

Possibly an inevitability, since LLM text is specifically generated to be low entropy. It’s smooth, optimized for minimal perplexity.

And if you don’t find any of the word choices a little bit perplexing, your brain can just gloss over it. There is no meaning here. It’s like smalltalk.

Post reply on HN