Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

41–50 of 382 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#42

Post by gitpusher42 »

mmastrac wrote: post #30

I have a project that's almost ready to run DiffusionGemma as well. The two project might potentially work well together. I'm getting ~20tok/s on a 36GB M3 and there's strong possibility we might be able to crib faster kernels from each other. Feel free to reach out. (currently at https://github.com/mmastrac/diffgemma but not in a releasable state yet)

It is super cool! Diffusion Gemma was released around the middle of my project, and I seriously considered switching to it. But I decided to finish the project as it was.

I believe it would be a perfect match!

Feel free to use any parts of my project or drop me a message. There’s my LinkedIn link at the end of the readme. Or I will drop you a message later!

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#45

Post by tracker1 »

Schiendelman wrote: post #39
addaon wrote: post #3

> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro. Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...

Not only is it older, so Pro:Pro it would have much slower SSD, but doesn't the Air also have slower SSD than the Pro in the same generation? And maybe narrower memory bandwidth?

IIRC, depends on the SSD size.. larger sizes had 2x the bandwidth, so it depends. That's combined/offset with the M5 improvements even further.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#46

Post by gitpusher42 »

owaislone wrote: post #43

Exciting! Maybe techniques like these can enable systems with 30-60GB memory and very fast SSDs of the future run very large models hopefully.

Yeah! Check the Colibri and Flash-MoE projects. They’re already doing that.

https://github.com/danveloper/flash-moe https://github.com/JustVugg/colibri

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#47

Post by fghorow »

sscarduzio wrote: post #17

How does this compare to DwarfStar4?

I'm curious too!

One obvious thing is that the memory requirements for this are substantially smaller than DwarfStar-- which AFAIK can only start to be used at 64GB ram and upwards. Another obvious thing is that antirez is pretty obsessed with making sure that DwarfStar passes all of DeepSeek V4 Flash's generating tests (loosely). I suspect that is also true of DwarfStar's implementation of GLM5.2, but I don't use that.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#48

Post by xenonite »

With my M1 MBA, I am still on macOS 15. To compile it, just remove the two lines with

  opts.languageVersion = .version4_0
or surround them with

  if #available(macOS 26.0, *) {
    opts.languageVersion = .version4_0
  }
You'll miss out on a prefill speedup of 2.4x (as it yields 11.24x faster attention), according to the git comments, but it works. (On the 8-GPU-core MBA M1, I get 5-6 tok/s.)

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#49

Post by gitpusher42 »

rcarmo wrote: post #44

Would be awesome if it ran Qwen (the MoE probably won't squeeze that low, but...). This because I have hardly been able to use Gemma for any sort of useful coding.

You’re right, Gemma isn’t the best model for coding (afaik more "everyday tasks" related). My first idea was to use Qwen, but its architecture was much more complex to implement in this stack. I chose Gemma so I wouldn’t spend all my time debugging custom kernels and could actually move the project forward with simpler approach

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#50

Post by a-dub »

gitpusher42 wrote: post #33
tredre3 wrote: post #20

I'm curious how your project compares to plain mmap! Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled). It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.

My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread` With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work And common weights still use mmap for simplicity So, I believe llama.cpp m…

for a given expert, do you have a sense for what the spatiotemporal access pattern looks like?
Post reply on HN