Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

github.com

Post reply on HN
Print view View original

101–110 of 126 posts

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#101

Post by monster_truck »

prometheus1992 wrote: post #31

It's hard to believe 16GB unified memory will give you 5 tok/sec unless you are ignoring the thermal warnings. I am running Qwen3.6-35B-A3B on my 16GB M3 and get 7-8 tokens/sec with all the optimizations while keeping the peak memory and thermal warnings at check. https://github.com/deepanwadhwa/samosa-chat

The laptops definitely can't hang but the minis don't really care. I threw mine down in the basement just to put the heat somewhere else, can tell when the dehumidifer next to it is on because it's a few C lower but that has no impact on performance. I don't think it's ever seen anything north of 70

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#102

Post by sersi »

mannyv wrote: post #69
AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

I think multiple people working on the same thing is great. Everyone comes at it from a different point of view, and some approaches work, some don't. And when people do this themselves they learn. Existing projects have their mistakes worked out already. Maybe one of these people is going to come up with the thing that nobody else thought of because of their experience working the problem from scratch. You may not g…

Multiple people working on the same thing is great. I'm less convinced about multiple people asking the same LLMs to redo the same project and coming up with a repo with a llm slop readme full of "Disk bites first", it's not this, it's that etc...

I don't think that this will bread any innovations.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#105

Post by jwr »

For what it's worth, I've spent some time with Claude to develop a local runner for `llama.cpp`. I run Qwen3.6-35B-A3B-MTP (fast!) and Qwen3.8-27B (20 tokens/s).

This was definitely worth the effort. Benchmarking and testing various approaches and various options really paid off. For example, one thing that surprised me was that MTP made things slower, not faster for Qwen3.8-27B.

I use a 64GB MacBook Pro (M4 Max).

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#106

Post by nixon_why69 »

jwr wrote: post #105

For what it's worth, I've spent some time with Claude to develop a local runner for `llama.cpp`. I run Qwen3.6-35B-A3B-MTP (fast!) and Qwen3.8-27B (20 tokens/s). This was definitely worth the effort. Benchmarking and testing various approaches and various options really paid off. For example, one thing that surprised me was that MTP made things slower, not faster for Qwen3.8-27B. I use a 64GB MacBook Pro (M4 Max).

I find mtp=3 does well with that model, only at 4 it becomes unprofitable.

Check your quants, its worth having the mtp layer be a bigger quant if it leads to 2x throughput from more accepted tokens.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#107

Post by MayeulC »

jmward01 wrote: post #93

Not a mac/UMA discussion point, but is it time to add additional, installable, DDR5 to GPUs? I can see this as a win/loose. PCIe 5x16 is close to maxing out the bandwidth available from high end dual channel DDR5 now, but not quite. I'm not a hardware person but I suspect putting it on the card could lead to significant performance improvements over using system ram so allowing systems like this, where MOE weights ar…

Probably not with DIMM modules, as longer traces mean higher latency (speed of light is ~30 cm in 1ns). GDDR typically uses larger buses (more wires) for higher bandwidth, even more so for HBM, so DIMM would be hard. Maybe CAMM would be up to the task?

It certainly seems feasible from an engineering perspective (though it does make cooling harder), at least for mid-range, not H100-class HW, but it prevents market segmentation, so EOMs may not be too interested (as long as no competitor does it).

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#108

Post by embedding-shape »

trollbridge wrote: post #75
embedding-shape wrote: post #21

> Hugging Face is the bottleneck, not your link. README could clearly make use of a cleanup, seems to be more like a session log dump now than a good introduction to the project for a new user. Maybe try something like "Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all flu…

I don’t want to be a cranky codger, but I dont get why 5 minutes of work cleaning up the README can’t be done before posting to HN.

Honestly? Me neither, I'd be embarrassed if I got caught not even looking at my own work before shipping it publicly.

However, feels like the battle is loosing, and now it's just about softening the landing instead of being vehemently against something the vast majority seems to be engaging in. I refuse to participate myself, but at least hopefully I could help steer some of them in a better (more concise) direction.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#109

Post by ch_sm »

jwr wrote: post #105

For what it's worth, I've spent some time with Claude to develop a local runner for `llama.cpp`. I run Qwen3.6-35B-A3B-MTP (fast!) and Qwen3.8-27B (20 tokens/s). This was definitely worth the effort. Benchmarking and testing various approaches and various options really paid off. For example, one thing that surprised me was that MTP made things slower, not faster for Qwen3.8-27B. I use a 64GB MacBook Pro (M4 Max).

I’m not an expert, but my understanding is that MTPs are smaller LLMs fine-tuned to "mimic" / predict a specific model’s response. It’s possible that the MTP you’re using isn’t trained well enough on Qwen 3.8. What accept rate are you getting?

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#110

Post by nixon_why69 »

ch_sm wrote: post #109
jwr wrote: post #105

For what it's worth, I've spent some time with Claude to develop a local runner for `llama.cpp`. I run Qwen3.6-35B-A3B-MTP (fast!) and Qwen3.8-27B (20 tokens/s). This was definitely worth the effort. Benchmarking and testing various approaches and various options really paid off. For example, one thing that surprised me was that MTP made things slower, not faster for Qwen3.8-27B. I use a 64GB MacBook Pro (M4 Max).

I’m not an expert, but my understanding is that MTPs are smaller LLMs fine-tuned to "mimic" / predict a specific model’s response. It’s possible that the MTP you’re using isn’t trained well enough on Qwen 3.8. What accept rate are you getting?

For qwen, it's an additional transformer layer at the very back, it ships as part of the model.
Post reply on HN