Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses • Hacker News Board

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#141

Post by Dylan16807 » Wed, Sep 09, 2026, 2:22 PM UTC

spider-mario wrote: post #137

Dylan16807 wrote: post #134

Earlier quoted context omitted.

I can see the analogy here but a test like that being binary throws things off and that's also super asymmetrical error. Weren't we sampling a gaussian?

It’s the same principle. The “trivial interval” from the first link (the one you called pedantry), which is applicable to a Gaussian and tweakable to have different coverage than 50%, clearly shows that being an n% confidence interval, on its own, doesn’t guarantee the ability to directly translate that n% into post-data inferences. Likewise, in the truncated exponential example, when you compute that confidence inte…

The trivial example demonstrates that once you pick your confidence interval you don't truly have odds anymore. But if we're judging whether we have the right odds number, what matters is if a different number is correct. And the only coherent way I can think of to interpret "odds" says the odds are 50%. The trivial example survives that challenge.

The examples that actually ruin the number so far have been asymmetrical. Is there a way to do it with a gaussian? Especially if you're trying to make a reasonable internal?

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#142

Post by anyfoo » Wed, Sep 09, 2026, 10:30 PM UTC

sharmajai wrote: post #139

anyfoo wrote: post #66

Earlier quoted context omitted.

Not for me. As stated elsewhere, even Q5 (!) seems to be enough to kill the model’s ability to solve a particular problem in reasonable time. But that might just be right at the edge of what the model can do in the first place. I have another personal benchmark problem (of a very different nature) that Qwen3.8-27B usually can’t solve at all, while Opus and GLM-5.3-Flash solve it consistently and very beautifully.

Without knowing anything about your benchmark, it might be your harness at fault here. I say this because with longer thinking you run the risk of filling up your context faster and you need a good context compaction strategy in your harness to mitigate that. I have heard good things about Pi which supports auto-compaction, but I can't personally vouch for it since I use my own.

You're probably partly right (though I use the same context size for all models, so there still is a difference in the model/quantization itself), but I definitely see it "thinking itself into an extremely expensive and error-prone approach without backing out". A common "failure mode" of my benchmark with Especially because Qwen3.8 27B does not seem to be good enough to implement the simulator with all its intricacies and tiny subtleties correctly. So it's churning for hours with no real progress, where it just had to explore the initial problem space a tiny bit more.

And very interestingly, even with Q6, turning off reasoning entirely does not make it fall into that trap, or many others, and it more often solves the task in record time. Both because of what I just described, but also because the endless reasoning costs a lot of token, which significantly translates to time on a home rig...

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#143

Post by sharmajai » Thu, Sep 10, 2026, 12:21 AM UTC

anyfoo wrote: post #142

sharmajai wrote: post #139

Earlier quoted context omitted.

Without knowing anything about your benchmark, it might be your harness at fault here. I say this because with longer thinking you run the risk of filling up your context faster and you need a good context compaction strategy in your harness to mitigate that. I have heard good things about Pi which supports auto-compaction, but I can't personally vouch for it since I use my own.

You're probably partly right (though I use the same context size for all models, so there still is a difference in the model/quantization itself), but I definitely see it "thinking itself into an extremely expensive and error-prone approach without backing out". A common "failure mode" of my benchmark with Especially because Qwen3.8 27B does not seem to be good enough to implement the simulator with all its intricaci…

I see, maybe your use case is indeed pathological for Qwen 3.8 27b. Another thing to try if you are using llama.cpp is "reasoning budget". That makes the thinking stop after the budget has been reached and inserts a custom message you can choose, so something like "you have thought for too long, now continue with the execution ..."

For what it's worth, if you haven't already, you can also let it run overnight (if you have compaction enabled) to see if it ever gets out of that hole. The reason `xhigh` is the default is because 3.8 is trying to optimize for long horizon tasks where monitoring its every single thinking misstep might not be a good use of our time.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#144

Post by nateb2022 » Thu, Sep 10, 2026, 3:39 AM UTC

> For comparison, DeepSeek V4 Flash 0731, a 284B model, costs around $0.1/Mtok for output from the cheapest providers on OpenRouter. I am not sure how much of this difference comes from the efficiency of running models at scale, pricing strategy, or popularity.

Due in large part to DeepSeek's MoE architecture. Generating 1 token through DS4's 13B active params requires roughly half the FLOPs to generate a token through the dense Qwen 27B.

Re: Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

#145

Post by mitxela » Thu, Sep 10, 2026, 5:39 AM UTC

spider-mario wrote: post #127

mitxela wrote: post #116

Earlier quoted context omitted.

> The key confusion underlying the FCF is the confusion of what is known before observing the data — that the CI, whatever it will be, has a fixed chance of containing the true value — with what is known after observing the data. Frequentist CI theory says nothing at all about the probability that a particular, observed confidence interval contains the true value; it is either 0 (if the interval does not contain the…

It’s not reaching. See my response here: https://news.ycombinator.com/item?id=49622516 > Like saying the probability your uncle died of a shark attack is either 0 or 1. For a frequentist, it is. If you want the Bayesian interpretation of probability, you must calculate the probability that the interval you ended up getting contains the true value, and you can easily end up with a value different from 95%. You can’t j…

If I pick an interval uniformly from a pool of intervals of which 95% contain the true value, it has a 95% chance of containing the true value.

Just like if I pick a ball from a bag of balls of which 50% are red, it has a 50% chance of being red.