Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

Post by sdlion » Thu, Jul 30, 2026, 3:52 AM UTC

spot5010 wrote: post #234

Keyframe wrote: post #168

Earlier quoted context omitted.

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

The human analogy would be that I don’t remember everything in the books I have read, but I do recall reading a particular book and can always look it up. So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.

Post by mmastrac » Thu, Jul 30, 2026, 4:00 AM UTC

nl wrote: post #261

mmastrac wrote: post #30

I have a project that's almost ready to run DiffusionGemma as well. The two project might potentially work well together. I'm getting ~20tok/s on a 36GB M3 and there's strong possibility we might be able to crib faster kernels from each other. Feel free to reach out. (currently at https://github.com/mmastrac/diffgemma but not in a releasable state yet)

I tried out one of the NVidia diffusion models, and from memory it only worked on MLX but seemed to leave a lot of features out. Would your work support non-Gemma models too?

Post by jtbaker » Thu, Jul 30, 2026, 4:22 AM UTC

Joeri wrote: post #132

tpurves wrote: post #55

Okay this tidbit is interesting to me "5–6 tok/s M2 -> 31–35 tok/s on an M5 Pro". So where will be in just another gen or two? my impression right now is that M5 gen is on the cusp of practicality for local inference. If techniques like OPs here, start to make the RAM situation more amenable, by the time we get to M6 or M7 (or AMD's equiv next gen APUs on TSMC N2 nodes), local AI could be ready to go much more mainst…

Without fundamental model architecture improvements the practicality largely depends on how Apple increases memory bandwidth. Memory bandwidths (* = rumored): M1: 68 GB/s M2: 100 GB/s M2 pro: 200 GB/s M2 max: 400 GB/s M2 ultra: 800 GB/s M5: 153 GB/s M5 pro: 307 GB/s M5 max: 460 GB/s M6: 200 GB/s* M7: 240 GB/s* Nvidia 4090 1008 GB/s Nvidia H100 3.35 TB/s Basically what we're looking at by the M7 generation is a tier s…

Post by jchook » Thu, Jul 30, 2026, 4:26 AM UTC

quasarj wrote: post #260

xenonite wrote: post #48

With my M1 MBA, I am still on macOS 15. To compile it, just remove the two lines with opts.languageVersion = .version4_0 or surround them with if #available(macOS 26.0, *) { opts.languageVersion = .version4_0 } You'll miss out on a prefill speedup of 2.4x (as it yields 11.24x faster attention), according to the git comments, but it works. (On the 8-GPU-core MBA M1, I get 5-6 tok/s.)

Why are you still on 15?

Post by jz391 » Thu, Jul 30, 2026, 4:33 AM UTC

bastawhiz wrote: post #190

giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

> I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to…