Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

github.com

Post reply on HN
Print view View original

61–70 of 126 posts

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#61

Post by dofm »

carloslfu wrote: post #44
dofm wrote: post #35

Earlier quoted context omitted.

I'm only joking anyway — it's more a comment on the whole AI-accelerated trend of everyone having their own version of a thing. I do agree that, ultimately, combining your efforts with others working in this whole area is probably really worth it, but I can see how there's an ease of pushing forward on your own these days. I do not have fast internet so I am not sure when I'll really be able to download the weights b…

I see! yes, downloading the weights part is painful. I tried a couple fixes and it is as fast as it can get downloading from HuggingFace. I think the field is heading toward smaller, more capable models soon, so you won't have to wait that long!

I mostly use Muse Glimmer, which runs quite well on my M1 Max as it is 30B but it also reasons very efficiently. I have tried the Qwen 3.8 27B which is also usable if quite slow to reason, but I guess like everyone the idea of the Flash-Next model holds some intrigue, because the 35B-A3B is pretty good.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#62

Post by red_hare »

atif089 wrote: post #42

As someone who is just looking at the theoretical benchmarks of each of these models I'm curious if anyone could share what are the problems (maybe around code) that flash-next was able to solve which 27b was not able to

For a local non-coding agent, instruction following and tool use are the most important gains

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#65

Post by prometheus1992 »

Balooga wrote: post #55
prometheus1992 wrote: post #31

It's hard to believe 16GB unified memory will give you 5 tok/sec unless you are ignoring the thermal warnings. I am running Qwen3.6-35B-A3B on my 16GB M3 and get 7-8 tokens/sec with all the optimizations while keeping the peak memory and thermal warnings at check. https://github.com/deepanwadhwa/samosa-chat

Now I'm feeling pretty good about getting 10-11 tokens/sec running Qwopus 3.6-35B-A3B Q6_K on an old Mac Pro 2013 (trashcan) with 128GB RAM (DDR3), 12 core Xeon, dual D700s. Arch Linux and llama.cpp.

haha, good for you.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#66

Post by jacquesm »

I love these efforts to get proper models running on lower cost hardware and I think this is where the next real breakthrough will come from. The more efficient this sort of thing can be done the bigger the chance to democratize this tech, 'good enough' is what you need and as long 'top of the line' gives a competitive edge even if it is at a cost there is a substantial risk of the door closing on general computing at some point in the near future. Keep in mind that there is no guarantee that the pendulum has to swing back, it can swing one way and get stuck, and then you're going to have to beg for crumbs from the haves.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#67

Post by cyanydeez »

Eufrat wrote: post #50
carloslfu wrote: post #37

Earlier quoted context omitted.

I feel you! fix incomming

For what it’s worth, this comment was not targeted at you, but rather the model kinda forcing it. I get the sense that Anthropic did not think much of this, but it seems to have gotten worse with recent models and it really comes off as a kind of nails on the chalkboard writing style. I have to image whatever style of writing this was trained on is a lot more pleasant to read and I feel bad for whoever writes like th…

most of this stuff isn't really possible at this point _without_ models, so pissing and moaning about people publishing work and not humanizing is really silly.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#69

Post by mannyv »

AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

I think multiple people working on the same thing is great.

Everyone comes at it from a different point of view, and some approaches work, some don't. And when people do this themselves they learn. Existing projects have their mistakes worked out already.

Maybe one of these people is going to come up with the thing that nobody else thought of because of their experience working the problem from scratch. You may not get that from someone working from an existing project, because existing projects have their approach "baked in."

What all these projects are showing so far is that it's possible to stream from disk, but that the performance isn't ideal. But I'm sure you could take this approach with smaller models and get better performance.

In addition, it's a given that when you work with large data sets performance means organizing the data to take advantage of caches, both disk and cpu. It's not clear how that would work, exactly, given that each run is a not-quite-random walk through the data. The Big Data way is to prebuild all of that as much as possible, which is probably impossible with a big model. But what about a smaller model?

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#70

Post by makira »

embedding-shape wrote: post #21

> Hugging Face is the bottleneck, not your link. README could clearly make use of a cleanup, seems to be more like a session log dump now than a good introduction to the project for a new user. Maybe try something like "Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all flu…

I used similar prompts before. Now I simply say to "remove historical cruft" and results are good enough. It's the model itself that first used this wording, I found it concise.
Post reply on HN