Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

31–40 of 382 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#31

Post by petu »

gitpusher42 wrote: post #23
afzalive wrote: post #7

Earlier quoted context omitted.

The M5 MBP has 24GB of RAM, more context in RAM perhaps?

The process stays at around 2 GB with 16 slots and a 4K context on both the M5 and M2. But yeah, Apple might be doing some magic under the hood

Unused RAM is wasted RAM. So not really Apple magic, about every OS uses "free" memory as disk cache.

Try to leave only a gigabyte or two free, speed likely would drop dramatically.

Edit: or do some calculation / logging of experts read speed, to see if it's faster than SSD spec.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#32

Post by maxignol »

I'm really excited about what's been happening couple last weeks for local inference. I feel like it all started after colibri [1] was released. Great work !

Anyone got recommendation about what local model to use for what purpose ? I feel like (as they were saying in moonshot blog post [2]) each llm can be an expert in its own categories and with several small local we might get good coverage for decent usage, granted each one is specialized enough.

[1] : https://github.com/JustVugg/colibri [2] : https://fireworks.ai/blog/kimik3-fable

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#33

Post by gitpusher42 »

tredre3 wrote: post #20

I'm curious how your project compares to plain mmap! Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled). It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.

My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread`

With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work

And common weights still use mmap for simplicity

So, I believe llama.cpp might run it under 2gb, but I assume it will be slower

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#34

Post by eamag »

mmastrac wrote: post #30

I have a project that's almost ready to run DiffusionGemma as well. The two project might potentially work well together. I'm getting ~20tok/s on a 36GB M3 and there's strong possibility we might be able to crib faster kernels from each other. Feel free to reach out. (currently at https://github.com/mmastrac/diffgemma but not in a releasable state yet)

Cool project! I looked into it recently and thought that running diffusion models locally doesn't really make sense: https://eamag.me/2026/why-parallel-diffusion-llms-are-slow-o...

What are your thoughts on this?

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#37

Post by giobox »

gitpusher42 wrote: post #29
greggh wrote: post #22

It does exactly what it says it does. On my Mac mini M4 with 16GB of ram it is running at just over 5 tok/s. That jump from M4 to M5 is crazy.

What exact specs do you have? It might be because it's the 256 GB version. afaik, those versions have much slower memory bandwidth than the 512 GB models My friend tried it on an M4 MacBook Pro and got 25–27 tok/s

This is correct, the 256gb is substantially slower as uses fewer physical memory chips - less ability to read/write in parallel. The 512gb or larger models have substantially higher read/write rates, and typically performs 50-100 percent faster in benchmarks than the 256.

Was a primary factor in me buying a 512gb M4 Mac Mini, even though I planned to use large external SSD - I wanted faster spec boot volume.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#38

Post by gitpusher42 »

maxignol wrote: post #32

I'm really excited about what's been happening couple last weeks for local inference. I feel like it all started after colibri [1] was released. Great work ! Anyone got recommendation about what local model to use for what purpose ? I feel like (as they were saying in moonshot blog post [2]) each llm can be an expert in its own categories and with several small local we might get good coverage for decent usage, grant…

I think I first saw Flash-MoE (https://github.com/danveloper/flash-moe) in April. Huge respect to them, it was a big inspiration for this project!

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#39

Post by Schiendelman »

addaon wrote: post #3

> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro. Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...

Not only is it older, so Pro:Pro it would have much slower SSD, but doesn't the Air also have slower SSD than the Pro in the same generation? And maybe narrower memory bandwidth?

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#40

Post by tracker1 »

This is actually very similar to some ideas I've been having for a while... that having a smaller entry model that knows enough about "expert" models that themselves are smaller to hand work over to could be better/faster/lighter in terms of working through real problems vs the megalith ones we currently use. Highly distilled experts and coordination with a fallback mode to a larger model option.
Post reply on HN