Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s • Hacker News Board

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#121

Post by jedbrooke » Thu, Sep 03, 2026, 4:24 AM UTC

I tried this in my mac mini m2 16GB, unfortunately I have to use an usb disk for the model weights, and I’m getting 0.5 tok/s. Still, being able to run (heh maybe crawl is more accurate) a 100B model on this computer AT ALL is pretty cool.

I see disk maxing out at 400 MB/s, this disk should be able to hit 1GB/s (it hits that eg when verifying the check sum of the weights), so there might be some optimization to be done there (I’m guessing it’s because the weights access is not pure sequential reads but involves some randomness depending on which expert)

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#122

Post by securecloudgrou » Thu, Sep 03, 2026, 12:01 PM UTC

Would love to see metrics of model performance, comparison with oMLX/OLLAMA/others. Any tooling for local optimization on hardware.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#123

Post by securecloudgrou » Thu, Sep 03, 2026, 12:03 PM UTC

securecloudgrou wrote: post #122

Would love to see metrics of model performance, comparison with oMLX/OLLAMA/others. Any tooling for local optimization on hardware.

To be more clear, I see you have slotstream doctor --sim-ram N, but extended tooling and optimization for exact local hardware would add value (MTPLX has a nice interface for example).

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#124

Post by carloslfu » Thu, Sep 03, 2026, 1:34 PM UTC

siris9476 wrote: post #81

32GB dedicated to an N-gram table instead of a draft model is an unusual choice for speculative decoding — what made it win over the more common draft-model approach here?

It wasn't either/or, the N-gram table is part of Qwen itself and stays on disk. I’ve now added its 1.5GB MTP draft head too, it gets 86% acceptance and about 1.24× faster decoding on my 48GB Mac.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#125

Post by siris9476 » Thu, Sep 03, 2026, 2:55 PM UTC

carloslfu wrote: post #124

siris9476 wrote: post #81

32GB dedicated to an N-gram table instead of a draft model is an unusual choice for speculative decoding — what made it win over the more common draft-model approach here?

It wasn't either/or, the N-gram table is part of Qwen itself and stays on disk. I’ve now added its 1.5GB MTP draft head too, it gets 86% acceptance and about 1.24× faster decoding on my 48GB Mac.

Got it, thanks for clearing that up. 86% acceptance is solid — does it stay flat over longer generations, or drift with context length?

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#126

Post by jonplackett » Thu, Sep 03, 2026, 11:42 PM UTC

mulemisterX wrote: post #60

I have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?

How are you finding it when it gets that high anyway? I’ve got a 64gb Mac so the context _can_ go higher. But I find around 70-80k it goes a bit weird and starts arguing with itself and doesn’t seem to know who it is vs me anymore.