Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8 27B

huggingface.co

Post reply on HN
Print view View original

111–120 of 848 posts

Re: Qwen 3.8 27B

#111

Post by metadat »

NitpickLawyer wrote: post #34
scrlk wrote: post #8

Beats Opus 4.7 Max (w/ Claude Code) on DeepSWE (42.2 vs 40). Looks like Qwen's 27B models continue to pack some punch. Unsloth's GGUF quants are up: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

> Beats Opus 4.7 Max I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage. Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expecta…

How can you say this when you haven't even tried it yet? Is it just hypothetical vibes?

Re: Qwen 3.8 27B

#112

Post by expedited123 »

kamranjon wrote: post #53
expedited123 wrote: post #13

Kinda was expecting to see Gemma 4 26B in benchmark comparisons :(

Since Qwen 3.6 27b outperforms Gemma 4 26b in most benchmarks I'm not sure the value - also Gemma 26b is a MOE model whereas this is a dense model, so not typically direct competitors at their sizes - Gemma 4 31b comparison would be interesting though.

I see! Thanks.

Re: Qwen 3.8 27B

#113

Post by cyanydeez »

spmurrayzzz wrote: post #48
NitpickLawyer wrote: post #34

Earlier quoted context omitted.

> Beats Opus 4.7 Max I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage. Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expecta…

> They do not beat opus on real-world usage We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8. For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC. This is very much "real-world usage…

Let us know when you have Qwen vs Qwen comparison stats. As long as there's not a regression, that'd be awesome.

Re: Qwen 3.8 27B

#114

Post by zargon »

scrlk wrote: post #74
WithinReason wrote: post #31

Earlier quoted context omitted.

I wish each quant was benchmarked on the same tests as the original network so we could compare their performance

Unsloth publishes KL divergence numbers which measures how much the quantised probability distribution changes vs unquantised: https://unsloth.ai/docs/models/qwen3.8#quantization-analysis It's a bit bare at the moment, I assume they are going to add further detail later (eg comparison to other quants), similar to their other releases.

KL divergence is nothing close to a replacement for benchmarks. As flawed as benchmarks are, KL divergence is a barely useful signal. The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.

Re: Qwen 3.8 27B

#115

Post by altmanaltman »

NitpickLawyer wrote: post #34
scrlk wrote: post #8

Beats Opus 4.7 Max (w/ Claude Code) on DeepSWE (42.2 vs 40). Looks like Qwen's 27B models continue to pack some punch. Unsloth's GGUF quants are up: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

> Beats Opus 4.7 Max I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage. Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expecta…

"Benchmark is stupid" and "model beats model on benchmark" are two different things, though. The second one is objectively true regardless of your views on the first one, right? To expect everyone to share your opinion that benchmarks are stupid is pretty weird, and just saying "no" to an objective truth is the definition of delusion.

Re: Qwen 3.8 27B

#116

Post by spmurrayzzz »

cyanydeez wrote: post #113
spmurrayzzz wrote: post #48

Earlier quoted context omitted.

> They do not beat opus on real-world usage We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8. For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC. This is very much "real-world usage…

Let us know when you have Qwen vs Qwen comparison stats. As long as there's not a regression, that'd be awesome.

4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.

Re: Qwen 3.8 27B

#117

Post by [deleted] »

jjice wrote: post #108
piyh wrote: post #57

Qwen 3.6 is ~$2/m tok, 3.8 should be drop in replacement. Gemma 31B is $0.34/m tok. The price differential on these models is massive on openrouter.

Where do you see that? From what I can see on Open Router, Qwen 3.6 27B (the closest dense equivalent to Gemma 31) is $0.28/m. Am I missing something? https://openrouter.ai/qwen/qwen3.6-27b

[deleted]

Re: Qwen 3.8 27B

#118

Post by jedbrooke »

spwa4 wrote: post #102
jedbrooke wrote: post #28

I hope the bonsai team makes another 1bit quant of this model (or releases code/instructions on how to do it), using the Qwen3.6 27B on my 16GB mac mini has been wild . The 1bit quant feels like opus level… for the first couple turns. Then it has trouble eg switching from plan mode to act mode. This is mostly mitigated by starting a new session. (tbf this limitation is called out on the hf page) I saw unsloth has 1bi…

Sounds like you need to check what the max context is set to ...

100k is all the context I have ram for, this is with any auto-compact turned off. This is using Cline in vs code. I’m sure I could tune the system prompt and mode switching more to work better with this specific model, but I haven’t gone down the custom harness rabbit hole yet.

And this is also specifically for the 1bit quant version. I don’t think the fp8 or even fp4 versions have this issue, but I haven’t tried those much

Re: Qwen 3.8 27B

#119

Post by Topfi »

ramon156 wrote: post #69
hypfer wrote: post #68

Earlier quoted context omitted.

> I've settled on GLM-5.3 (formerly Deepseek v4 pro 0813) for architecting Dude, GLM-5.3 released _today_. The phrasing "I've settled on" is incorrect for this context.

hence the "former deepseek v4 pro". I tried it out this morning and have had no complaints. I already liked glm 5.2

Honest question, how do you assess models this quickly? What metrics are you using? Would love to get my suite from multiple days and hundreds of prompts down to minutes. Got a few first pass tasks I run upon release for an initial experience, but those only work because even Fable and Sol fail despite objectively correct solutions existing, so it works because most models fail, but then, those are consciously not enough for coding, tool use, adherence or task specific inference and assessment…

Re: Qwen 3.8 27B

#120

Post by seanmcdirmid »

TomGarden wrote: post #24

Any tips on the best approach at running this at an M4 Max 128GB? Token throughput was a bit slow with the last 27B one (MLX), ended up using the A3B variant but if I could get this one to reasonable speed I'd much prefer it.

27B is a dense model so it will be slower with an MoE (A3B), but should have better quality? I still haven’t found very good uses cases on my M3 Max for dense models. Even if you can find a MTP version, it doesn’t help much, especially if you compare against an MoE with MTP as well.
Post reply on HN