Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#1

Post by carloslfu » Tue, Sep 01, 2026, 4:42 PM UTC

I built slotstream, a way to run Qwen3.8-Flash-Next 4-bit on a low-memory mac starting from 16GB, a 125B parameter model that would need 100GB+ memory/RAM, thanks to expert-offloading/ssd-streaming. Easy to install/update, and mac-native using MLX and Swift.

It ships with auto-mode, which makes a good tradeoff between memory usage and speed. I'll be implementing and porting the MTP module for speculative decoding next

Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
github.com

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#2

Post by AmazingTurtle » Tue, Sep 01, 2026, 5:03 PM UTC

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon.

At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-LM instead of producing yet another near-identical repo.

The local-LLM ecosystem really does not need every implementation idea rediscovered five times and wrapped in a new README. AI-assisted coding makes producing a new repo cheap; maintaining, benchmarking, and integrating one is the actually valuable part.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#3

Post by api » Tue, Sep 01, 2026, 5:09 PM UTC

AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

> every implementation idea rediscovered five times and wrapped in a new README

That's open source since forever, unfortunately.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#4

Post by Barbing » Tue, Sep 01, 2026, 5:10 PM UTC

AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

Vouched especially since OP might have a perspective on this. And readers may want to look up those other repos and compare for themselves.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#5

Post by dofm » Tue, Sep 01, 2026, 5:15 PM UTC

AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

AI NIH

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#6

Post by carloslfu » Tue, Sep 01, 2026, 5:16 PM UTC

AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

I see your point. As an oss defender myself, I agree, however, the spirit of this is to see how fast I can make it. I'm sharing this with the community, which I think is aligned with the original oss spirit.

It's an experiment for myself but I am committing to maintain it. I've been an oss person for a loooong time, way before AI was a thing. Think about it as a new, from-scratch take at it, not as a re-reproduction.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#7

Post by carloslfu » Tue, Sep 01, 2026, 5:19 PM UTC

Barbing wrote: post #4

AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

Vouched especially since OP might have a perspective on this. And readers may want to look up those other repos and compare for themselves.

Thanks for the feedback! I'll create a section with a benchmark and comparisons. This will hold the project accountable and speed things up imo

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#8

Post by docheinestages » Tue, Sep 01, 2026, 5:19 PM UTC

api wrote: post #3

AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

> every implementation idea rediscovered five times and wrapped in a new README That's open source since forever, unfortunately.

It's what happens when you don't do market research.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#9

Post by carloslfu » Tue, Sep 01, 2026, 5:20 PM UTC

dofm wrote: post #5

AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

AI NIH

Sorry, I don't get "NIH". what's that?

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#10

Post by kzrdude » Tue, Sep 01, 2026, 5:21 PM UTC

AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

And there are `Mference` and `SwiftLM` too, I think they are doing the same use case.