Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

quesma.com

Post reply on HN
Print view View original

51–60 of 138 posts

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#51

Post by kennywinker »

vardalab wrote: post #47
Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

Just ask your freaking agent to read it for you and extract the information. That's what it's all about. Why would I be reading these articles other than information?

Claude, read Love in the Time of Cholera for me and summarize the information contained. You are an expert book reader and understander. Make no mistakes.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#52

Post by conmod278 »

seemaze wrote: post #40
sharmajai wrote: post #39

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level. It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the…

As they say, time is money. In the age of the rampocalypse, the peasants may not have a choice between the two.. time it is!

Computer science has known the tradeoffs between memory and compute since ages ago. The same could be reflected here.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#53

Post by lowbloodsugar »

cogman10 wrote: post #24
Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that. The line is "Is this an interesting and accurate article that concisely makes it's case". LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very…

Sure, but if you're going to publish it, at least run it through an edit prompt and tell it to remove clickbait "Its not X, its Y" rubbish. Like literally calling examples clickbait in the prompt has given me better results. Interestingly, I have a lot less trouble with the first draft with Qwen than with Opus.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#54

Post by chmod775 »

seemaze wrote: post #40
sharmajai wrote: post #39

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level. It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the…

As they say, time is money. In the age of the rampocalypse, the peasants may not have a choice between the two.. time it is!

Smaller models are also generally faster, so thinking "more" may not matter and may even come out ahead.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#55

Post by stared »

spider-mario wrote: post #4

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…

Point taken, but there is a much more fundamental issue with it - and precisely why I wrote "very conservative".

It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.

Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas.

That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#56

Post by maCDzP »

spider-mario wrote: post #4

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…

Thank you for these, coz I learned a lot! Great that they are open access.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#57

Post by kennywinker »

dotinvictim wrote: post #27

local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable

It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#58

Post by alentred »

I would be very interested in a similar benchmark for *KV cache* quantizations.

I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there.

I also heard that KV cache quantization matters more with longer contexts. It may be interesting to benchmark this too: what the quality looks like on different combinations of model quantization × KV cache quantization × context size.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#59

Post by dofm »

cogman10 wrote: post #37
dofm wrote: post #31

Earlier quoted context omitted.

My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem. In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.

Perhaps it's just a bias? You are already negatively biased against LLM writing so you disregard stuff you read when you suspect it's an LLM. This could also be a selection bias. It may be that you generally struggled to retain information but you are more aware of it when LLMs are involved.

I don't think so, no, because it extends to LLM-generated text I want to recall.

I use LLMs to generate starter/tutorial material. I may hate the way Claude writes but I absolutely don't hate the way Gemma 4 writes. But I have to continuously consult it in a way that I do not with human-written text, which gets its message across in a more persistent way I find less troubling.

(This non-memorability extends to AI images and video.)

You are right that there are some confounding factors in my life but while I was worried about middle age affecting recall, I find actually I am still remembering stuff humans wrote pretty much fine.

ETA: there is one thing that I have noticed that does affect recall that is specific to LLMs: watching text roll out word-by-word in LLM chat, I think, damages recall. It's engaging the wrong kind of memory and focus.

So I now let it generate and I'm trying to find decent ways to format it e.g. as PDF, to give it the best chance.

This could be generational; there were many studies twenty years ago that suggest that people a decade or so younger than me who grew up with full colour books and magazines and multimedia can read less-linear text layouts more comfortably, for example, so it's not out of the question that there's something generational going on here too.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#60

Post by celrod »

chmod775 wrote: post #54
seemaze wrote: post #40

Earlier quoted context omitted.

As they say, time is money. In the age of the rampocalypse, the peasants may not have a choice between the two.. time it is!

Smaller models are also generally faster, so thinking "more" may not matter and may even come out ahead.

If Q4 takes less than 1.3x as many tokens as bf16 or q8, it could still end up being faster, given how decode tends to be bandwidth bound. The kv cache was still bf16, so a few ops are the same between quants.
Post reply on HN