32GB dedicated to an N-gram table instead of a draft model is an unusual choice for speculative decoding — what made it win over the more common draft-model approach here?
Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
81–90 of 126 posts
Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
#82How usable is 12 tok/s?
Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
#83Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
#84Is this going to destroy my SSD?
A particularly worrying situation considering a dead SSD will render your macbook usable for parts only.
Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
#85I have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?
[dead]
Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
#86How usable is 12 tok/s?
Unpleasant for interactive agentic work. Still useful to leave it to do some work in the background.
Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
#87Earlier quoted context omitted.
Vouched especially since OP might have a perspective on this. And readers may want to look up those other repos and compare for themselves.
Thanks for the feedback! I'll create a section with a benchmark and comparisons. This will hold the project accountable and speed things up imo
Absolutely! Nice.
(The comments under the parent indicate it was improperly flagged/made dead (maybe could happen just from downvoting?) so glad I hit the Vouch.)
Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
#88> Hugging Face is the bottleneck, not your link. README could clearly make use of a cleanup, seems to be more like a session log dump now than a good introduction to the project for a new user. Maybe try something like "Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all flu…
> Run Qwen3.8-Flash-Next on a Mac that can't hold it
This is the first line of the README. I can't believe people are becoming ok with this, and I'm 100% on the AI train.
Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
#89I have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?
We are running 35b-A3b with 264k context (the model's default max) using vllm and the "frog" jinja templates: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates and had good luck. We are mostly running agentic workloads though, rather than coding. 27b has a slightly higher agentic job completion rate (95% vs 92%) but the 3% trade off is worth it because the A3B is sooooo much faster, and we reprocess the other jobs with a different model. Don't sleep on the froggeric templates.
Qwen: Looking at you for a new ~35B MoE! Please and thank you
Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
#90Is this going to destroy my SSD?
no it's reading, not writing
Using macos on low memory regimes will make it use disk-based swap.
For example, a Macbook Neo (so in theory, something with around 4GiB of free RAM lying around) might eat around 900GB of writes a day while not doing much at all, because it's basically on low on RAM and swapping all the time.