Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

221–230 of 383 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#221

Post by gitpusher42 »

dznodes wrote: post #216

Please explain what is useful about this repo to me as if I were a high school dropout. Is this like claude.ai but running on my own hardware? Does it need to be on the internet to be useful? Do I need CS skills to install and use it?

hm. just open repo, copy commands into your terminal and you will get app installed (if you have swift toolchain installed)

after that download 14gb of weights and enjoy offline inference (and a bit of Gemma4 intelligence) for your everyday tasks

multi turn chat is coming!

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#222

Post by dinkelberg »

rpdillon wrote: post #205
evilduck wrote: post #202

Earlier quoted context omitted.

Because it's a strong signal of AI slop. Why put in more work than the "author" did? If the author generated text that required no effort, and has no understanding of the contents of the material generated, and no self-awareness of their behavior and how the audience will receive it, it definitely doesn't warrant wasting a single second reading it. Now, granted, maybe they did review it, maybe they did understand it,…

The term "AI slop" is thought-terminating. A more nuanced approach: read it and decide for yourself on merits, rather than vibes.

Life is short. Do you want to spend it reading text that was evidently generated by a machine? There is an opportunity cost to reading "slop."

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#223

Post by gitpusher42 »

n4pw01f wrote: post #219

Are you running bare metal or docker? I had to go down to 2 bit on Gemma4 E2B model to run on 8gb on a Jetson Quality and idempotency is great but it’s still not exactly fast… fast enough and works offline Is this something that you can get running on Debian?

It is Apple platform only implementation because of Metal (and Swift). Other platforms would require CUDA or Vulkan and a complete rework

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#224

Post by walrus01 »

giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

> I don't care who King Charles is every single time

There is just a stupendous amount of everything packed into a 35B size or larger model. For instance Qwen 3.6 35B A3B (Q8) can do a fairly decent job translating English to Arabic, but it can also generate python code with a reasonable layout and commenting.

I try to remember that as a mental model, an epub copy of a truly gargantuan sized 1000+ page novel such as the unabridged/2nd revision of Steven King's The Stand is about 800KB, and we're talking about a GGUF file that's 37GB in size or something like that.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#225

Post by e12e »

itsdesmond wrote: post #126
andrehacker wrote: post #107

Earlier quoted context omitted.

Let my karma burn for saying this: Maybe it is time to let this go man. These comments are really the new incarnation of "grammar policing". (1) They don't add anything of value, did the author use an LLM to fix his prose but no useless slop was added in the process: who cares ? Is the article useless slop: fine, downvote it to oblivion. (1) For those not old enough to remember that wonderful practice please use your…

The writing is made worse by a specific choice which the commentator identified. That’s actionable feedback.

I don't know what the author's native language is, but I assume it's something I'd need machine translation for anyway. Having it in good (if not Nobel prize level great) English is much easier - and will probably be easier to find again via search.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#226

Post by icedchai »

GeekyBear wrote: post #110
wongarsu wrote: post #8

Earlier quoted context omitted.

My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads If that's true, inference speed would be even lower if you have only 2GB total, including OS caches

The base level M5 doesn't just have more memory than the base level M2. The memory bandwidth is bumped up by 50%, and the size of the on-die system level cache is bumped up by 50% as well.

I recently got an M5 Air as a second machine. I question if I'll even need a Pro machine in the future, as long as I can get enough RAM with the base chip (which seems unlikely, but...)

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#227

Post by Almondsetat »

JSR_FDED wrote: post #145
andrehacker wrote: post #107

Earlier quoted context omitted.

Let my karma burn for saying this: Maybe it is time to let this go man. These comments are really the new incarnation of "grammar policing". (1) They don't add anything of value, did the author use an LLM to fix his prose but no useless slop was added in the process: who cares ? Is the article useless slop: fine, downvote it to oblivion. (1) For those not old enough to remember that wonderful practice please use your…

Why would you want to signal low effort for your writing and the related project?

The world is not the Unites States

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#228

Post by gitpusher42 »

jeffybefffy519 wrote: post #211

Kind of interesting, really what experts do is sort/organise weights into categories that are optimal to work together. Seems like a lot of research could be done to extend this concept to group weights together for common inputs ahead of time to achieve the same purpose.

Yeah, I tried both rearranging experts on disk and predicting the next expert using statistical approach. Reordering helped on the test prompt, but failed on another prompt. Markov and cross layer prediction didn't work either

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#230

Post by whatsThisBtn4 »

DAE read these CPU Mac posts as an example of

https://en.wikipedia.org/wiki/Reality_distortion_field

I own an Nvidia chip and even then I find these models fast but not useful for contemporary AI.

I can't imagine slow and useless.

It reminds me of that US politician that has controlled the minds of 30% of the population.

Post reply on HN