Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

quesma.com

Post reply on HN
Print view View original

31–40 of 137 posts

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#31

Post by dofm »

cogman10 wrote: post #24
Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that. The line is "Is this an interesting and accurate article that concisely makes it's case". LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very…

My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem.

In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#34

Post by exogenousdata »

jnwatson wrote: post #10
spider-mario wrote: post #4

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…

Mind blown. The more I read about statistics, the less I know.

“There are three kinds of lies: Lies, damned lies and statistics.” - Mark Twain (attributed but unsubstantiated to Benjamin Disraeli)

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#35

Post by Farmadupe »

cogman10 wrote: post #24
Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that. The line is "Is this an interesting and accurate article that concisely makes it's case". LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very…

(If it helps, I ask my own question of myself too -- I mostly don't write code by hand any more as I find that an LLM writes it faster and with less bugs -- Is that therefore proof that my time was never worth my paycheck? I hope not but at the same time I would actually be proud if I had got away with being an accidental charlatan/fraudster at my employer's expense during my entire career)

-----

Similarly, if what I said really is true, I would be implying that LLMs are charlatan/fraudster detectors (to some statistical level). And I refuse on principle to believe that that is actually the case.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#36

Post by cogman10 »

JSR_FDED wrote: post #29
cogman10 wrote: post #24

Earlier quoted context omitted.

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that. The line is "Is this an interesting and accurate article that concisely makes it's case". LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very…

Except that wasting the reader’s time became a lot easier with LLMs.

Yes, LLMs make it a lot easier to produce a lot more garbage. That's not an exception to my point. If something is well written then it doesn't waste the reader's time.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#37

Post by cogman10 »

dofm wrote: post #31
cogman10 wrote: post #24

Earlier quoted context omitted.

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that. The line is "Is this an interesting and accurate article that concisely makes it's case". LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very…

My main problem — which I am sure being middle-aged compounds — is that I struggle to retain information that an LLM has written or produced. I cannot explain why but it is a consistent problem. In a week’s time I might remember the substance of your comment and some of its shape as a matter of course. Nothing LLM-written that I see today will stick, no matter how curated it was.

Perhaps it's just a bias? You are already negatively biased against LLM writing so you disregard stuff you read when you suspect it's an LLM. This could also be a selection bias. It may be that you generally struggled to retain information but you are more aware of it when LLMs are involved.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#39

Post by sharmajai »

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level.

It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the task (benchmark or otherwise).

So as long as the thinking (albeit longer) is sound, this leads to the same success rate (as shown in the article) but potentially at the cost of more tokens and hence more time.

I think it'll be further useful to chart each quantization's used tokens as well, in addition to the success rate.

Thanks for doing and sharing the research!

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#40

Post by seemaze »

sharmajai wrote: post #39

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level. It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the…

As they say, time is money.

In the age of the rampocalypse, the peasants may not have a choice between the two.. time it is!

Post reply on HN