Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

github.com

Post reply on HN
Print view View original

41–50 of 126 posts

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#41

Post by xlayn »

carloslfu wrote: post #6
AmazingTurtle wrote: post #2

There are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-…

I see your point. As an oss defender myself, I agree, however, the spirit of this is to see how fast I can make it. I'm sharing this with the community, which I think is aligned with the original oss spirit. It's an experiment for myself but I am committing to maintain it. I've been an oss person for a loooong time, way before AI was a thing. Think about it as a new, from-scratch take at it, not as a re-reproduction.

Hey carloslfu, kudos from the other side of the internet, don't get down on people nitpicking everything here, experimenting and discovering is part of learning so keep going!, remember this is the place that said dropbox was dumb and could be replaced by a script.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#44

Post by carloslfu »

dofm wrote: post #35
carloslfu wrote: post #24

Earlier quoted context omitted.

Ah! Yeah, I didn't invent anything (yet!). The goal is to see how far I can take it in terms of speed without consuming that much RAM.

I'm only joking anyway — it's more a comment on the whole AI-accelerated trend of everyone having their own version of a thing. I do agree that, ultimately, combining your efforts with others working in this whole area is probably really worth it, but I can see how there's an ease of pushing forward on your own these days. I do not have fast internet so I am not sure when I'll really be able to download the weights b…

I see! yes, downloading the weights part is painful. I tried a couple fixes and it is as fast as it can get downloading from HuggingFace. I think the field is heading toward smaller, more capable models soon, so you won't have to wait that long!

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#46

Post by embedding-shape »

xlayn wrote: post #39
embedding-shape wrote: post #21

> Hugging Face is the bottleneck, not your link. README could clearly make use of a cleanup, seems to be more like a session log dump now than a good introduction to the project for a new user. Maybe try something like "Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all flu…

[flagged]

> if someone completely new comes and read the current page... isn't that piece of information something they want to know?

Again, the README is (was?) written in way that it was clear it was a conversation between an agent and a human, with literal replies in the README. I'm in no way whining the way you are or demonstrate, and I even give OP helpful advice to fix it. What more can you want from me?

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#50

Post by Eufrat »

carloslfu wrote: post #37
Eufrat wrote: post #34

Earlier quoted context omitted.

I hate this AI style writing because since it doesn’t really understand flow, it’s being inserted in irrelevant places and it is extremely irritating to read.

I feel you! fix incomming

For what it’s worth, this comment was not targeted at you, but rather the model kinda forcing it. I get the sense that Anthropic did not think much of this, but it seems to have gotten worse with recent models and it really comes off as a kind of nails on the chalkboard writing style.

I have to image whatever style of writing this was trained on is a lot more pleasant to read and I feel bad for whoever writes like this now being associated as bad AI writing.

Post reply on HN