Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

71–80 of 382 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#71

Post by Aurornis »

tpurves wrote: post #55

Okay this tidbit is interesting to me "5–6 tok/s M2 -> 31–35 tok/s on an M5 Pro". So where will be in just another gen or two? my impression right now is that M5 gen is on the cusp of practicality for local inference. If techniques like OPs here, start to make the RAM situation more amenable, by the time we get to M6 or M7 (or AMD's equiv next gen APUs on TSMC N2 nodes), local AI could be ready to go much more mainst…

I have an M5 128GB. Being on the cusp of practical is a good description. It will run, but prefill and token gen are still slow relative to my consumer GPU box.

It also gets very hot. If you’ve never heard the fans on Apple Silicon really spin up, it could surprise you. Makes the full GPU setup feel quiet by comparison.

I think after the hardware market calms down the ticket is going to be a light laptop with a second dedicated inference server on the network.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#72

Post by nijave »

gitpusher42 wrote: post #33
tredre3 wrote: post #20

I'm curious how your project compares to plain mmap! Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled). It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.

My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread` With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work And common weights still use mmap for simplicity So, I believe llama.cpp m…

Any idea if madvise helps? Admittedly I have very limited experience and only on Linux

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#73

Post by mandeepj »

hacklas wrote: post #67
owaislone wrote: post #43

Exciting! Maybe techniques like these can enable systems with 30-60GB memory and very fast SSDs of the future run very large models hopefully.

How large? With 64 GB of unified memory, you should be able to run a DeepSeek V4 Flash quantisation at 7–10 t/s, for example with: https://github.com/antirez/ds4 or https://github.com/steadfastgaze/MoEspresso (my engine). The routed experts needed for the next tokens that are not already in memory need to be read from the SSD, so the speed becomes SSD reading bound and the larger the memory, the faster the inference.

> 7–10 t/s

Maybe use it for overnight batch work! Hopefully, you aren’t suggesting it using for realtime conversations!

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#75

Post by Barbing »

ambicapter wrote: post #28
GeekyBear wrote: post #14

Earlier quoted context omitted.

The M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation. > In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift. https://www.tomshardware.com/laptops/…

Extremely impressive.

Not sure Data (Star Trek TNG) could read that fast.

“Accessing”

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#77

Post by greggh »

gitpusher42 wrote: post #29
greggh wrote: post #22

It does exactly what it says it does. On my Mac mini M4 with 16GB of ram it is running at just over 5 tok/s. That jump from M4 to M5 is crazy.

What exact specs do you have? It might be because it's the 256 GB version. afaik, those versions have much slower memory bandwidth than the 512 GB models My friend tried it on an M4 MacBook Pro and got 25–27 tok/s

Yeah, its the 256gb version.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#79

Post by mft_ »

hacklas wrote: post #67
owaislone wrote: post #43

Exciting! Maybe techniques like these can enable systems with 30-60GB memory and very fast SSDs of the future run very large models hopefully.

How large? With 64 GB of unified memory, you should be able to run a DeepSeek V4 Flash quantisation at 7–10 t/s, for example with: https://github.com/antirez/ds4 or https://github.com/steadfastgaze/MoEspresso (my engine). The routed experts needed for the next tokens that are not already in memory need to be read from the SSD, so the speed becomes SSD reading bound and the larger the memory, the faster the inference.

Is there a particular quant of DS v4 Flash you'd recommend that works on 64GB machines? None of the antirez versions on HF look small enough?

Also, FWIW, I've been experimenting with Laguna-S-2.1. It runs reasonably quickly (llama.cpp, IQ2_M quant) but the outputs so far aren't impressive, and it gets stuck and perseverates. Very subjectively, at that level of quantisation, it seems to perform worse than Qwen 3.6 27B at Q4_K_XL.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#80

Post by [deleted] »

a-dub wrote: post #50
gitpusher42 wrote: post #33

Earlier quoted context omitted.

My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread` With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work And common weights still use mmap for simplicity So, I believe llama.cpp m…

for a given expert, do you have a sense for what the spatiotemporal access pattern looks like?

[deleted]
Post reply on HN