Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

github.com

Post reply on HN
Print view View original

91–100 of 126 posts

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#91

Post by kamranjon »

mulemisterX wrote: post #60

I have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?

I am running 3.8 27b at q6 quant with 160k context on a 32gb video card (arc b70 pro) - I quantized the kv cache at q8 - that is the only trick really - works great.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#93

Post by jmward01 »

Not a mac/UMA discussion point, but is it time to add additional, installable, DDR5 to GPUs? I can see this as a win/loose. PCIe 5x16 is close to maxing out the bandwidth available from high end dual channel DDR5 now, but not quite. I'm not a hardware person but I suspect putting it on the card could lead to significant performance improvements over using system ram so allowing systems like this, where MOE weights are shed, to get even higher performance than just adding that DDR5 to the system. Bigger models become closer to reality and it provides more of a pathway for developing technologies that take advantage of it. Of course the loose side is that you just put a lot of specialized ram on a card instead of into the system where it could be used for other things. I could see a place for a 16GB card with 64GB(or more) of DDR5 especially if we start seeing MOE and similar technologies really start being designed for this concept.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#94

Post by carloslfu »

brcmthrowaway wrote: post #59
brailsafe wrote: post #57

Earlier quoted context omitted.

This is one of the aspects of this year that I've been finding very grating and wasteful. Collaboration still happens among people with the ability to do so and the technical skills, but everyone else is taking their own helicopter to the top of the mountain, "putting it out there", and there's just a ton of redundant projects that do the same thing.

It's horrible. Every 20-something working on a load-bearing inference engine on GitHub.

for the record, I'm almost 35

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#95

Post by carloslfu »

mannyv wrote: post #69
AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

I think multiple people working on the same thing is great. Everyone comes at it from a different point of view, and some approaches work, some don't. And when people do this themselves they learn. Existing projects have their mistakes worked out already. Maybe one of these people is going to come up with the thing that nobody else thought of because of their experience working the problem from scratch. You may not g…

true

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#97

Post by carloslfu »

c0rruptbytes wrote: post #92

so many inference project, omlx already supports all of this and has a 1000 people trying to optimize it constantly

Both projects are different in scope. Think of slotstream as optimizing for memory and for this specific model for now, my intention is not to build an inference engine the same as oMLX

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#98

Post by brailsafe »

carloslfu wrote: post #94
brcmthrowaway wrote: post #59

Earlier quoted context omitted.

It's horrible. Every 20-something working on a load-bearing inference engine on GitHub.

for the record, I'm almost 35

Also for the record, I wasn't trying to take a personal shot at you or your project—nor am I sure of how valid that would be, if that were to have been my intention—it's just a gripe I have in general what I think is somewhat degrading the trust I can have in certain types of projects, especially those that someone shits out, puts on the app store, appears on the surface to visibly look ok, but ultimately has no uniquely valuable contribution or long-term outlook and is just someone's zero to one replication of something that has an api and already exists. It is admittedly cynical, but I now scrutizinize what I pay for more aggressively as a consequence.
Post reply on HN