Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

291–300 of 382 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#291

Post by sznio »

pwython wrote: post #245

Ran this on a 64 GB M4 Max MacBook. I figured having Gemma available with a small footprint would be a nice setup. No more unloading models when I need more RAM for work? Hell yea. Got 48 tok/s decode at 1.9 GB RSS (2.4 GB peak), faster than the 24 GB M5 Pro mentioned in the benchmarks. The ~2.0 GB/s SSD number quoted for M4 is the base chip. This M4 Max does ~7 GB/s. Page cache seems to be why it beats the M5 Pro. W…

at that much ram you can just load it outright without tricks. it will be much faster even if it ends up swapping.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#293

Post by dznodes »

gitpusher42 wrote: post #221
dznodes wrote: post #216

Please explain what is useful about this repo to me as if I were a high school dropout. Is this like claude.ai but running on my own hardware? Does it need to be on the internet to be useful? Do I need CS skills to install and use it?

hm. just open repo, copy commands into your terminal and you will get app installed (if you have swift toolchain installed) after that download 14gb of weights and enjoy offline inference (and a bit of Gemma4 intelligence) for your everyday tasks multi turn chat is coming!

what is a swift tool chain?

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#295

Post by harrouet »

fouc wrote: post #249
pwython wrote: post #245

Ran this on a 64 GB M4 Max MacBook. I figured having Gemma available with a small footprint would be a nice setup. No more unloading models when I need more RAM for work? Hell yea. Got 48 tok/s decode at 1.9 GB RSS (2.4 GB peak), faster than the 24 GB M5 Pro mentioned in the benchmarks. The ~2.0 GB/s SSD number quoted for M4 is the base chip. This M4 Max does ~7 GB/s. Page cache seems to be why it beats the M5 Pro. W…

M4 Max is typically better than M5 Pro for inference IIRC.

It depends on what you are looking at.

Time to 1st token is faster on the M5 because of HW accelerators helping the prompt interpretation (and it is CPU-bound).

Token generation after that is GPU-bound and will profit from the higher bandwidth of the M4 Max.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#296

Post by code51 »

giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

You don't know whether King Charles or 42 is the answer to "the Question"

my 2c on King Charles vs 42.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#298

Post by gitpusher42 »

pwython wrote: post #245

Ran this on a 64 GB M4 Max MacBook. I figured having Gemma available with a small footprint would be a nice setup. No more unloading models when I need more RAM for work? Hell yea. Got 48 tok/s decode at 1.9 GB RSS (2.4 GB peak), faster than the 24 GB M5 Pro mentioned in the benchmarks. The ~2.0 GB/s SSD number quoted for M4 is the base chip. This M4 Max does ~7 GB/s. Page cache seems to be why it beats the M5 Pro. W…

Thank you very much for sharing! Great results and useful info!

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#299

Post by [deleted] »

jrgifford wrote: post #248
gitpusher42 wrote: post #12

Earlier quoted context omitted.

I haven't tried it but it should work! You can try it and share your results, it would be really appreciated I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s Also, it must be easy to adjust for iPhones and iPads in theory

Confirmed on my Neo! Got 4.5 tokens / second sustained.

[deleted]

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#300

Post by gitpusher42 »

jrgifford wrote: post #248
gitpusher42 wrote: post #12

Earlier quoted context omitted.

I haven't tried it but it should work! You can try it and share your results, it would be really appreciated I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s Also, it must be easy to adjust for iPhones and iPads in theory

Confirmed on my Neo! Got 4.5 tokens / second sustained.

Thank you for testing and sharing, it is useful info!
Post reply on HN