Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

quesma.com

Post reply on HN
Print view View original

21–30 of 137 posts

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#21

Post by Farmadupe »

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"?

Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc),

But on the other hand, the result that I'm assuming the author means to communicate ("on these evals, generation quality seems fairly good") sounds worthwhile to share?

Because I really struggle with this question at the moment. Am I allowed to draw an adverse inference that "if the writeup presents irrelevant text side by side with the data, then this may be a sign that the author does not understand the task that they are attempting to write up"?

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#22

Post by sroussey »

bellowsgulch wrote: post #5

Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it. Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work th…

Have you tried https://huggingface.co/prism-ml/Bonsai-27B-mlx-1bit ? PrismML is the only people i am aware of doing 1bit that is decent.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#23

Post by clircle »

Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

I think the advice is the same regardless of AI use: read articles written by authors that have a history of high quality writing.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#24

Post by cogman10 »

Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that.

The line is "Is this an interesting and accurate article that concisely makes it's case".

LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very little information feel more substantial.

I say, stop trying to determine if an LLM was used and start judging based on your subjective measure that you'd have used before LLMs became widespread.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#25

Post by [deleted] »

fr2029 wrote: post #16
spider-mario wrote: post #4

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower. Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them ( https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run varia…

the 2nd derivate of shannon covariance of noise begs to differ

[deleted]

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#26

Post by dofm »

purpleflame1257 wrote: post #2

There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.

There is an interesting new dynamic 3 bit quantisation I have been meaning to test:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

Luke of Luke’s Dev Lab on YouTube had a look at it. It seems to outperform the typical 3-bit quantisation but whether it outperforms the new Unsloth dynamic I don’t know.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#28

Post by dofm »

jadbox wrote: post #11
purpleflame1257 wrote: post #2

There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.

Q3 XL and Q3 XS are the two I'm trying to decide on

You might want to test this new dynamic GGUF:

https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF

(I don’t know much about it, just saw a YouTube video about it last night)

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#29

Post by JSR_FDED »

cogman10 wrote: post #24
Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

IMO, whether or not an LLM was used in the writing process doesn't really matter and I think it's a bit annoying that articles are being dismissed out of hand because of that. The line is "Is this an interesting and accurate article that concisely makes it's case". LLMs love to burn paragraphs writing about nothing which is why it's generally poor writing. Humans can do the same thing if they are trying to make very…

Except that wasting the reader’s time became a lot easier with LLMs.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#30

Post by JSR_FDED »

Farmadupe wrote: post #21

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"? Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc), But on the other hand, the result that I'm assuming the author means to communicate ("on these eva…

You don’t need anyone’s permission. You have only so much attention, why spend it wading through slop?
Post reply on HN