Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

11–20 of 382 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#11

Post by gitpusher42 »

yakupov_bulat wrote: post #2

Wow, amazing! What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine.

It depends on the use case.

I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s.

The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit.

If you need maximum speed and flexibility just use MLX

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#12

Post by gitpusher42 »

znpy wrote: post #6

I wonder if i can run this on my MacBook Neo!

I haven't tried it but it should work! You can try it and share your results, it would be really appreciated

I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s

Also, it must be easy to adjust for iPhones and iPads in theory

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#14

Post by GeekyBear »

addaon wrote: post #3

> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro. Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...

The M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation.

> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.

https://www.tomshardware.com/laptops/macbooks/m5-macbook-pro...

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#18

Post by gitpusher42 »

h2aichat wrote: post #10

Hope you can do it for Windows users also (and small graphics cards). Thanks

Uh, I’m afraid it is Apple only. It is written using Apple’s GPU language, Metal, and heavily relies on the Apples’s shared memory architecture

Windows PCs would require a completely different approach

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#19

Post by hsienchuc »

I've run local video generation models on an 8GB graphics card and know firsthand that nothing runs smoothly when memory is insufficient. So seeing 14GB of weights crammed into 2GB of RAM is impressive.

If running continuously for over an hour (like an overnight batch task), will a fanless MacBook Air overheat and throttle? Can the SSD handle the continuous weight reads and sustained output speeds?

Great work, congratulations on the release!

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#20

Post by tredre3 »

I'm curious how your project compares to plain mmap!

Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled).

It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.

Post reply on HN