Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac • Hacker News Board

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#241

Post by oblio » Thu, Jul 30, 2026, 12:25 AM UTC

giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

> Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and practicality, they just leave it to whoever sets up infrastructure to worry about.

The idea that people in 10+ frontier labs (OpenAI, Anthropic, Google, Alibaba, Z.ai, DeepSeek, xAI, Amazon, etc) in a trillion dollars industry are all dumb is frankly, hilarious.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#242

Post by Gigachad » Thu, Jul 30, 2026, 12:32 AM UTC

spot5010 wrote: post #234

Keyframe wrote: post #168

Earlier quoted context omitted.

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

The human analogy would be that I don’t remember everything in the books I have read, but I do recall reading a particular book and can always look it up. So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.

LLMs can do that. The one that's built in to google search is relatively stupid, but it just uses web pages to fill in the gaps.

The downside is it is incredibly easy to be tricked by a single false bit of info.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#243

Post by dh303 » Thu, Jul 30, 2026, 1:14 AM UTC

[dead]

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#244

Post by supportm » Thu, Jul 30, 2026, 1:27 AM UTC

very cool, thanks for sharing

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#245

Post by pwython » Thu, Jul 30, 2026, 1:31 AM UTC

Ran this on a 64 GB M4 Max MacBook. I figured having Gemma available with a small footprint would be a nice setup. No more unloading models when I need more RAM for work? Hell yea.

Got 48 tok/s decode at 1.9 GB RSS (2.4 GB peak), faster than the 24 GB M5 Pro mentioned in the benchmarks. The ~2.0 GB/s SSD number quoted for M4 is the base chip. This M4 Max does ~7 GB/s.

Page cache seems to be why it beats the M5 Pro. With 64 GB the whole 12 GB packed_experts set stays resident, and iostat shows only ~1.6 GB per run actually reaching disk, against the ~79 GB that 98 fully cold tokens would need.

I then tested with DaVinci Resolve open and under load (playback): 42.6 tok/s. Also held 38 GB of incompressible memory to squeeze the page cache: 41.8. At 48 GB it ranged 32 to 41.5. Degrades gradually rather than a cliff. It's a beautiful thing.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#246

Post by JSR_FDED » Thu, Jul 30, 2026, 1:37 AM UTC

jryle70 wrote: post #203

JSR_FDED wrote: post #145

Earlier quoted context omitted.

Why would you want to signal low effort for your writing and the related project?

It's not low effort. The author had to write in English, not their native, and then used LLM to polish it. The sentence itself conveyed a real point. They cared how their article came across. That's much more effort than the boring "Claude was here" comment that took a second to write but costs real energy to appear, and once again spurred a worthless debate. Let it go FFS.

I hear you, the author put in real effort. That’s why it’s a tragedy if it presents like low effort.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#247

Post by troyzhxu » Thu, Jul 30, 2026, 1:40 AM UTC

spot5010 wrote: post #234

Keyframe wrote: post #168

Earlier quoted context omitted.

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

The human analogy would be that I don’t remember everything in the books I have read, but I do recall reading a particular book and can always look it up. So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.

[flagged]

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#248

Post by jrgifford » Thu, Jul 30, 2026, 1:43 AM UTC

gitpusher42 wrote: post #12

znpy wrote: post #6

I wonder if i can run this on my MacBook Neo!

I haven't tried it but it should work! You can try it and share your results, it would be really appreciated I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s Also, it must be easy to adjust for iPhones and iPads in theory

Confirmed on my Neo!

Got 4.5 tokens / second sustained.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#249

Post by fouc » Thu, Jul 30, 2026, 2:00 AM UTC

pwython wrote: post #245

Ran this on a 64 GB M4 Max MacBook. I figured having Gemma available with a small footprint would be a nice setup. No more unloading models when I need more RAM for work? Hell yea. Got 48 tok/s decode at 1.9 GB RSS (2.4 GB peak), faster than the 24 GB M5 Pro mentioned in the benchmarks. The ~2.0 GB/s SSD number quoted for M4 is the base chip. This M4 Max does ~7 GB/s. Page cache seems to be why it beats the M5 Pro. W…

M4 Max is typically better than M5 Pro for inference IIRC.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#250

Post by kcexn » Thu, Jul 30, 2026, 2:15 AM UTC

giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

I don't think the problem is that memory footprint can't be reduced. The problem is that the token generation rate is simply far too low.

OP suggests that the token rate of of their solution is ~5 per second. That's at least an order of magnitude slower than commercially available models.