Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

github.com

Post reply on HN
Print view View original

71–80 of 127 posts

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#71

Post by mrob »

cromka wrote: post #45
jonplackett wrote: post #43

Is this going to destroy my SSD?

By reading it?

Reading causes insignificant wear ("read disturb") that likely isn't a problem, but I don't think it's possible to issue pure reads to modern SSDs. The NVMe spec mandates tracking the amount of data read, and this has to be written to the drive. I'd hope the firmware buffers this and writes it at low frequency, but on the other hand, I doubt the firmware was tested in extreme random-read regimes. Unexpected failures from excessive statistics recording could be possible.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#72

Post by ig0r0 »

mulemisterX wrote: post #60

I have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?

yes, with qwen3.8-27b-4bit run via rapid-mlx i can get to about 200k

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#73

Post by metadat »

ig0r0 wrote: post #72
mulemisterX wrote: post #60

I have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?

yes, with qwen3.8-27b-4bit run via rapid-mlx i can get to about 200k

Nice.. and how many toks/sec?

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#74

Post by cosmic_cheese »

jacquesm wrote: post #66

I love these efforts to get proper models running on lower cost hardware and I think this is where the next real breakthrough will come from. The more efficient this sort of thing can be done the bigger the chance to democratize this tech, 'good enough' is what you need and as long 'top of the line' gives a competitive edge even if it is at a cost there is a substantial risk of the door closing on general computing a…

I think there's a very good chance that history will rhyme a bit.

DOS/Windows and PC clones were by no means the best available, but they were cheap, ubiquitous, and versatile compared to alternatives that were either much better at one task but more expensive or better at everything but wildly expensive. They were "good enough" and represented a solid improvement over what many existing computer users had as well as a good entry point for new users. As such they spread like wildfire and became the standard while the expensive alternatives either became hardcore niche or vanished.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#75

Post by trollbridge »

embedding-shape wrote: post #21

> Hugging Face is the bottleneck, not your link. README could clearly make use of a cleanup, seems to be more like a session log dump now than a good introduction to the project for a new user. Maybe try something like "Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all flu…

I don’t want to be a cranky codger, but I dont get why 5 minutes of work cleaning up the README can’t be done before posting to HN.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#76

Post by trollbridge »

whartung wrote: post #26

I'm hoping to see progress in this space. Folks talking about how 32G is not enough for local use, but then there's been work like this to empower it. My hope is that the new 32G M6 will be "useful" locally, possibly because of work like this.

32GB is simply too tight; you need 8 minimum for the OS and you need about 4-8 more for the LLM you’re visiting and kv cache.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#77

Post by trollbridge »

prometheus1992 wrote: post #31

It's hard to believe 16GB unified memory will give you 5 tok/sec unless you are ignoring the thermal warnings. I am running Qwen3.6-35B-A3B on my 16GB M3 and get 7-8 tokens/sec with all the optimizations while keeping the peak memory and thermal warnings at check. https://github.com/deepanwadhwa/samosa-chat

Anything smaller than a 16” runs into serious thermal problems; even an identically equipped 14” just can’t dissipate enough heat.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#78

Post by jacquesm »

cosmic_cheese wrote: post #74
jacquesm wrote: post #66

I love these efforts to get proper models running on lower cost hardware and I think this is where the next real breakthrough will come from. The more efficient this sort of thing can be done the bigger the chance to democratize this tech, 'good enough' is what you need and as long 'top of the line' gives a competitive edge even if it is at a cost there is a substantial risk of the door closing on general computing a…

I think there's a very good chance that history will rhyme a bit. DOS/Windows and PC clones were by no means the best available, but they were cheap, ubiquitous, and versatile compared to alternatives that were either much better at one task but more expensive or better at everything but wildly expensive. They were "good enough" and represented a solid improvement over what many existing computer users had as well as…

SUN Apollo SGI

Though to be fair it was Linux more than Windows that killed them. Dos and Windows were competition for DEC and - ironically - IBM.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#79

Post by pram »

mulemisterX wrote: post #60

I have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?

You should try Glimmer MTP. Qwen3.8 27B seems to have weird memory and caching issues on oMLX

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#80

Post by rzzzt »

0x457 wrote: post #18
karmakaze wrote: post #15

It seems we could use a new kind of memory that streams the weight data in, like GDDR in reverse.

High Bandwidth Flash? https://www.sandisk.com/company/newsroom/blogs/2025/scaling-...

Optane? (Too soon?)
Post reply on HN