Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

modelscope.cn

Post reply on HN
Print view View original

151–160 of 178 posts

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#151

Post by cpburns2009 »

htrp wrote: post #148
cpburns2009 wrote: post #80

Earlier quoted context omitted.

This is the best I got, all with Unsloth's quantizations. Laguna-S-2.1:UD-Q4_K_XL (no MTP) pp=186.4 t/s tg=27.8 t/s Qwen3.6-35B:UD-Q4_K_XL (with MTP) pp=404.4 t/s tg=83.2 t/s Qwen3.6-27B:UD-Q4_K_XL (recorded pre-MTP) pp=343 t/s tg=12.1 t/s Laguna actually performed better than I remembered. I thought it was slower.

Have you benchmarked against full precision models for accuracy/ performance?

Not full precision. I've only benchmarked 27B across Q3-6 quants using lm-eval. I lack the hardware to bench 27B at BF16 but I might be able to do Q8_0. I haven't gotten around to doing 35B. I really should upload my collection of results to Github or somewhere.

Here's a summary of what I have for 27B. I used unsloth's UD-Q{3-6}_K_XL quants across 11 evals. The values are pretty linear between Q3 and Q6.

    Qwen3.6-27B     Q3    Q6
    ARC-Challenge   97.0  97.0
    BIG-Bench Hard  57.9  59.3
    GPQA Diamond    77.8  83.3
    GSM8K           92.4  92.6
    Hendrycks Math  35.5  38.9
    HumanEval       80.5  85.4
    HumanEval+      75.0  79.3
    IFEval          87.3  88.0
    MBPP            75.2  77.2
    MBPP+           88.4  88.9
    MMLU-Pro        83.1  83.5

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#152

Post by jubilanti »

SwellJoe wrote: post #108
jubilanti wrote: post #94

Earlier quoted context omitted.

A 3060ti 8gb, released in 2020, has 448 GB/s of bandwidth compared to the Halo 256 GB/s The 3080ti is 912.4 GB/s

Enjoy Gemma 4 E2B at blistering speeds, I guess?

My point was more: this was the 2nd lowest end card from a generation 6 years ago, and it had way higher bandwidth than today's alleged flagship.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#153

Post by ActorNightly »

embedding-shape wrote: post #98
jubilanti wrote: post #94

Earlier quoted context omitted.

A 3060ti 8gb, released in 2020, has 448 GB/s of bandwidth compared to the Halo 256 GB/s The 3080ti is 912.4 GB/s

And the newly announced/launched Apple M6 has 170GB/s of unified memory bandwidth, meanwhile M5 Ultra gets 1.2TB/s of unified memory bandwidth. https://www.apple.com/newsroom/2026/08/apple-introduces-m6-a... Not sure if the first one is a typo on their press release, can't be just 170GB/s then be pushed for AI use, can it? Could be a different measurement I suppose...

GFX vram is still faster.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#154

Post by ActorNightly »

jubilanti wrote: post #94
SwellJoe wrote: post #40

Finally, a reason to own a 128GB Strix Halo or GB10 device. Or a reason to consider the new Mac Studio. I have a Strix Halo and dual 32GB GPUs in my desktop, and the latter is pretty much always better for running local models because it's quite a bit faster due to higher memory bandwidth. There simply haven't been any models that are better than Qwen 27B or Gemma 31B, which run comfortably in 64GB with big context.…

A 3060ti 8gb, released in 2020, has 448 GB/s of bandwidth compared to the Halo 256 GB/s The 3080ti is 912.4 GB/s

Arguing with people on here that you should buy GFX cards instead of overpriced Macs for inference is a lost cause. Either Apple astroturfs this forum hard enough to convince people Macs are good for local llm, or people are REALLY stupid and don't understand how local inference works and think that the dogshit slow 40-50 tok/sec is standard.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#156

Post by big-chungus4 »

culi wrote: post #145
big-chungus4 wrote: post #10

> We are releasing these architectural improvements ahead of time so that the community can prepare for the upcoming full family of Qwen4 models. That gives me hope that "full family" means it will include smaller models like 4B.

What are the use-cases for a model as small as 4B?

They are great base models for fine tuning on both text and visual tasks. Many OCR and object grounding models are based on small Qwen models, though they often replace vision encoder with a bigger one.

Qwen3 5-4B is the biggest model I can find tune in my laptop. And when I upgraded the model from Qwen3-4B to Qwen3.5-4B, both vanilla and fine tuned performances jumped significantly on a classification task.

Those models are great when you have very little data or very low diversity of examples, where it's not possible to train a neural net from scratch as it will just memorize the data. The best you can do is fine tune a generalist model that can already do the task for small number of steps until it starts over-fitting, or on some cases you can do even better though RL.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#158

Post by latentsea »

hasteg wrote: post #22
cogman10 wrote: post #7

Wow. I wasn't expecting this. I thought they were going to do a 35B model instead.

As a 5090 owner and local model enthusiast, I was hoping it would be 35B A3B so I could run it myself =(.

Depends on the rest of your hardware, and how the the n-gram weights work and if they can be streamed from SSD. If they can and you have 64GB system ram then you should actually be able to run it.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#159

Post by latentsea »

_ache_ wrote: post #120
embedding-shape wrote: post #118

Earlier quoted context omitted.

How long is a rope? Technically you could probably run it off a SSD, but it'll be slow as molasses. If you want it "fast", you want it all within GPU and VRAM, who knows what that'd be. If the engram parameters are separate, I guess it'd be like BF16 ~400 GB, FP8 ~200 GB, NVFP4 ~100GB. Otherwise maybe like ~300GB, ~150GB and ~70GB or alike, don't quote me that, only some guesses. The one who waits will see :)

I think a reasonable expectation of MAX requirement to claim "runable on consumer hardware" is to 32G VRAM and 128GB RAM and it run at +10tps.

I've got an R9700 32GB and an RTX 5060 Ti 16GB plus 64GB of system ram. Hoping to be able to run this at around 30 t/s on a Q4 quant. Hoping. Really hoping. Anything below that isn't usable as a daily driver since at deep context it drops quite significantly, so if you start out at say 20 t/s then you'll wind up at like 10 t/s and 20 t/s is already too slow.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#160

Post by latentsea »

hgoel wrote: post #147
rdsubhas wrote: post #137

Earlier quoted context omitted.

How do you folks code at 40-50 tps? With an extremely lightweight harness (pi) and just 8k system and tools context, and ~40tps on qwen 3.8 27B 4-bit on low thinking mode, it still takes me nearly 30-45 mins for a basic coding session... Does it work? yeah... But I'd pick a subscription anyday...

Do you find subscriptions to be meaningfully faster? I didn't really feel too much of a speed difference compared to Opus.

As someone who uses Opus daily for professional work and Qwen3.8-27B for all my private stuff, yes, Opus sub is faster for me, but I'm only rocking an R9700. If you're lucky enough to have sold a kidney on the blackmarket and purchased a 5090 and you're running ninfer, then actually... I think you'd be seeing fairly comparable performance!
Post reply on HN