It will be interesting to see the intersection of this with inference engines like FreeToken which improve distribution of work for MoE models across CPU/RAM and GPU/VRAM. If all it takes for a competitive model to run locally at good speeds is a used 3090 and some DDR4, then we might be in for the year of local AI. https://github.com/FlashML-org/FreeToken
Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
81–90 of 178 posts
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#82Finally, a reason to own a 128GB Strix Halo or GB10 device. Or a reason to consider the new Mac Studio. I have a Strix Halo and dual 32GB GPUs in my desktop, and the latter is pretty much always better for running local models because it's quite a bit faster due to higher memory bandwidth. There simply haven't been any models that are better than Qwen 27B or Gemma 31B, which run comfortably in 64GB with big context.…
how much performance (tok/s) can you expect from 128GB Strix Halo? assuming this model will be released with FP8 also can you use it for fine tuning?
A model similar in size to Laguna S 2.1, but with only 6B active parameters, should be a notable amount faster, so I would imagine 25-30 t/s would be a reasonable guess for where Qwen 3.8 Flash Next will land.
DFlash2 might improve all these numbers. It wasn't available last I was testing new models on the Strix Halo; I've only used MTP (which doesn't generally improve MoE models, but I believe DFlash2 can).
Given software improvements, I'm hopeful an MoE in this size range will be the sweet spot that pushes past 40 t/s and is also smart enough for real work. Qwen 3.8 27B is finally a self-hostable model that's smart enough, but it thinks so hard it still isn't really useful for agentic interactive use.
Note also prefill with large models is pretty slow on the Strix Halo (300 t/s, maybe). Time to first token is a painful wait, when using it interactively with large models.
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#83I was already rolling around the idea of a 128GB M5 Max MBP. Now this! A 4-bit MLX quant with 128k window should fit perfectly, in the 50-70 tok/s range.
Just out of curiosity, why run "local-local" when you could just set up a Mini or Studio at home and query it over http? [edit] whole conversation about this in another thread https://news.ycombinator.com/item?id=49433413 I’m personally considering retiring my MBP for a Studio + 15" Air whenever this MBP ages out.
I’m doing that. Mac Mini M4 Pro with 48G RAM as a headless llama.cpp server.
I much prefer using " thin clients " as the interface to the big VMs running in my homelab
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#84Very curious to see how this compares to Deepseek v4 Flash. I have to assume they wouldn't be releasing this if it was worse.
Why not? It's not really competing in the same size class. Besides, as they explicitly wrote here, the main goal for this release is not performance, rather to serve as a reference for inference runtimes about what to implement. So that later Qwen 4 can be released with zero day support.
Guess Qwen 4 is the one to wait for.
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#85+1 to the long list of people hoping for Qwen3.8-27b A3B.
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#86Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#87Finally, a reason to own a 128GB Strix Halo or GB10 device. Or a reason to consider the new Mac Studio. I have a Strix Halo and dual 32GB GPUs in my desktop, and the latter is pretty much always better for running local models because it's quite a bit faster due to higher memory bandwidth. There simply haven't been any models that are better than Qwen 27B or Gemma 31B, which run comfortably in 64GB with big context.…
That’s only true if you think AI is the only reason to own a powerful and efficient server. Mine does plenty of traditional server stuff too.
Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#88Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#89Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)
#90Qwen4 sounds exciting