Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

modelscope.cn

Post reply on HN
Print view View original

141–150 of 178 posts

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#141

Post by wongarsu »

FuckButtons wrote: post #131
wongarsu wrote: post #68

Earlier quoted context omitted.

There is the rule of thumb that if you take the geometric mean of the total and active parameters of an MoE model you get the equivalent size of an equally capable dense model. If you follow that formula, you would expect a 125b-a6b model to match a 27b model (sqrt(125*6) = 27.3). That does not feel like a coincidence

Where does that rule of thumb come from?

That's a great question. I learned it on HN. Some searching around suggests it originated as an empirical observation in the local LLM space around 2023-2024

It's obviously just a rough approximation. Actual scaling laws suggested in published papers are a lot more complex, and even then you run into issues (architecture changes, effects like better training, putting intelligence on a one-dimensional axis is stupid in the first place, etc). But as an approximation it holds up pretty well for normal-ish ratios between active and total parameters

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#142

Post by mhast »

jtbayly wrote: post #123
embedding-shape wrote: post #98

Earlier quoted context omitted.

And the newly announced/launched Apple M6 has 170GB/s of unified memory bandwidth, meanwhile M5 Ultra gets 1.2TB/s of unified memory bandwidth. https://www.apple.com/newsroom/2026/08/apple-introduces-m6-a... Not sure if the first one is a typo on their press release, can't be just 170GB/s then be pushed for AI use, can it? Could be a different measurement I suppose...

You got me curious so I looked up the previous chips[0]. Memory bandwidth M1: 68 GB/s M2: 100 GB/s (47% increase) M3: 100 GB/s (0% increase) M4: 120 GB/s (20% increase) M5: 153 GB/s (27.5% increase) So, M6: 170 GB/s (11% increase) doesn’t seem impossible, though I would have expected more. [0]: https://www.jdhodges.com/blog/apple-cpu-compared-m1-m3-m3-m4...

The different models of chips and memory config have very different memory speeds as well.

Eg the M4 Max 128GB has a bandwidth speed of 500GB/s+. And that's true for other models as well.

But as you note, the base speed has also increased over the versions.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#143

Post by decide1000 »

SwellJoe wrote: post #140
decide1000 wrote: post #139

Earlier quoted context omitted.

On the DGX I get 44.5 tokens per second (NVFP4). With 8 concurrent it's 241 t/s total. I am using the PrismaAQUA standard 9.7 t/s + Dflash2 30 t/s + torch-compile 37 t/s c8 = 177 t/s

What model? Also, I don't know what "the PrismaAQUA" means, ddg thinks it's a CPAP machine, which seems unlikely to help with inference performance. Also, 4-bit has measurable intelligence loss. Sometimes worth it, but, at this size models are barely smart enough at 8 or 6.

Qwen3.8-27B-PrismaAQUA-5.5bit-vllm

The output quality is higher. It's held at full precision (not quantized).

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#144

Post by entrope »

DwarvenEngineer wrote: post #132
brandall10 wrote: post #127

Earlier quoted context omitted.

From a bandwidth perspective, the ultra is like 8 M5s fused together (@ 150GB/s), that's how it gets to the 1200. Historically the Pro doubles the base, the Max doubles the Pro, and the Ultra doubles the Max. If an M6 ultra were released today it would be 1.36TB/s.

does that mean they're measuring bandwidth differently than how others (like nvidia) does it? memory bandwidth is the gating factor of running models locally, so if it's actually 8x 150GB/s, it may help something like prefill, but would it actually speed up decode comparatively?

No, they use the same definition of memory bandwidth as others, but Apple Silicon has a lot of memory channels. In previous generations, prefill has been compute-limited and decode is fast.

https://blog.exolabs.net/nvidia-dgx-spark/ outlines a combination of a DGX Spark and an M3 Ultra that took advantage of fast prefill on the Nvidia hardware and fast decode on Apple Silicon.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#145

Post by culi »

big-chungus4 wrote: post #10

> We are releasing these architectural improvements ahead of time so that the community can prepare for the upcoming full family of Qwen4 models. That gives me hope that "full family" means it will include smaller models like 4B.

What are the use-cases for a model as small as 4B?

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#146

Post by polski-g »

irthomasthomas wrote: post #17
ddtaylor wrote: post #14

I enjoy the Qwen models a lot, but building things on top of them with OpenRouter has been painful. OpenRouter does a lot of great work and I really enjoy being able to use different models so easily. I like when a provider is phasing out an older model that still works for my needs and the price is much lower. It seems like such a good win-win. However, the problem is that many Qwen models have almost no capacity or…

Openrouter was pretty great before prompt caching became common. Now it is extremely expensive for most individual workflows, unless you spend a lot of work customizing router preferences, and then you still get a worse cache hit rate than using the provider directly. I only keep $5-$10 in OR for occasional testing.

Use the "only" header?

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#147

Post by hgoel »

rdsubhas wrote: post #137
pwython wrote: post #11

I was already rolling around the idea of a 128GB M5 Max MBP. Now this! A 4-bit MLX quant with 128k window should fit perfectly, in the 50-70 tok/s range.

How do you folks code at 40-50 tps? With an extremely lightweight harness (pi) and just 8k system and tools context, and ~40tps on qwen 3.8 27B 4-bit on low thinking mode, it still takes me nearly 30-45 mins for a basic coding session... Does it work? yeah... But I'd pick a subscription anyday...

Do you find subscriptions to be meaningfully faster? I didn't really feel too much of a speed difference compared to Opus.

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#148

Post by htrp »

cpburns2009 wrote: post #80
puzzlingcaptcha wrote: post #71

Earlier quoted context omitted.

What sort of pp/tg speed do you get on a Strix Halo?

This is the best I got, all with Unsloth's quantizations. Laguna-S-2.1:UD-Q4_K_XL (no MTP) pp=186.4 t/s tg=27.8 t/s Qwen3.6-35B:UD-Q4_K_XL (with MTP) pp=404.4 t/s tg=83.2 t/s Qwen3.6-27B:UD-Q4_K_XL (recorded pre-MTP) pp=343 t/s tg=12.1 t/s Laguna actually performed better than I remembered. I thought it was slower.

Have you benchmarked against full precision models for accuracy/ performance?

Re: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

#150

Post by vietvu »

ddtaylor wrote: post #14

I enjoy the Qwen models a lot, but building things on top of them with OpenRouter has been painful. OpenRouter does a lot of great work and I really enjoy being able to use different models so easily. I like when a provider is phasing out an older model that still works for my needs and the price is much lower. It seems like such a good win-win. However, the problem is that many Qwen models have almost no capacity or…

Who use openrouter for serious usage? Caching would be really bad unless you use preset to pin to single provider.
Post reply on HN