Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

331–340 of 382 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#331

Post by gitpusher42 »

petu wrote: post #31
gitpusher42 wrote: post #23

Earlier quoted context omitted.

The process stays at around 2 GB with 16 slots and a 4K context on both the M5 and M2. But yeah, Apple might be doing some magic under the hood

Unused RAM is wasted RAM. So not really Apple magic, about every OS uses "free" memory as disk cache. Try to leave only a gigabyte or two free, speed likely would drop dramatically. Edit: or do some calculation / logging of experts read speed, to see if it's faster than SSD spec.

yeah, looks like a page cache matters a lot I tested on mine m5 pro with 8gb memory pressure, got 27t/s instead of 35t/s Someone tested on m4 max. In regular state it was 48tok/s, but 32-42 with memory pressure

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#332

Post by anon373839 »

sznio wrote: post #290

Nice. Gemma feels nice to write with but every time I use it for coding it struggles with tool calling significantly.

Have you tried the new chat template Google released recently? It’s supposed to address this and enable reasoning content preservation. I have not tried it myself but am hoping it does the trick, since Gemma is great model otherwise.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#333

Post by myrmidon »

apothegm wrote: post #316
swiftcoder wrote: post #149

Earlier quoted context omitted.

> I don't care who King Charles is every single time Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

That’s the thing. LLMs don’t have any logical ability. Only predictive ability. They’re not the same. And that’s why LLMs are a) unreliable and b) not a path to AGI.

How do you know that human logic is any more/better/qualitatively different from the LLMs "predictive ability"? It's already pretty easy to find humans that are strictly worse at reasoning and logic than a decent LLM.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#334

Post by joefourier »

oblio wrote: post #327
joefourier wrote: post #325

Earlier quoted context omitted.

Anthropic's API has two nines availability and Claude Code is a TUI made with React that can regularly consume more than 1GB of RAM, and the codebase is utter slop. They couldn't fix the flickering bug for over a year! And yet, Fable and Opus are among the best coding models out there (matched only by GPT5.6 Sol). It's not about the people there being smart or not, it's about their and the company's priorities, resou…

One is the UX, which they don't care about because people use their models anyway. The second would be hardware savings on the order of tens of billions of dollars if they were stupid not do try all possible optimizations. Spot the difference.

You cannot just "try all possible optimisations". It takes time, effort, and money that could otherwise be spent elsewhere (especially for training, where each training run is especially costly, and optimisations might be promising early on, but cause the final performance of the model to be worse). You need smart people interested in unglamorous work, and if you're swimming in VC money, it's far more straightforward to just throw more GPUs at the problem.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#335

Post by admash »

myrmidon wrote: post #333
apothegm wrote: post #316

Earlier quoted context omitted.

That’s the thing. LLMs don’t have any logical ability. Only predictive ability. They’re not the same. And that’s why LLMs are a) unreliable and b) not a path to AGI.

How do you know that human logic is any more/better/qualitatively different from the LLMs "predictive ability"? It's already pretty easy to find humans that are strictly worse at reasoning and logic than a decent LLM.

[flagged]

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#336

Post by reddguard »

cwillu wrote: post #147
reddguard wrote: post #93

Earlier quoted context omitted.

Doesn't Ollama use llama.cpp so their point stands even if they used it directly?

It wouldn't be the first time ollama's llama.cpp fork reintroduced bugs and was missing important optimizations.

lol jealous hater

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#337

Post by swiftcoder »

anentropic wrote: post #308
swiftcoder wrote: post #149

Earlier quoted context omitted.

> I don't care who King Charles is every single time Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

I suspect there's a conceptual problem here to what extent is "retain the logical ability" meaningful without attaching it to some knowledge

> to what extent is "retain the logical ability" meaningful without attaching it to some knowledge

Some knowledge is obviously required, I'm just less sure that a specific task like coding benefits all that much from having Shakespeare in the training set...

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#338

Post by glitchc »

panarky wrote: post #273
glitchc wrote: post #255

Earlier quoted context omitted.

Indeed, it begs the question why we have "everything" models where instead we could have very efficient "something" models. Typical LLMs out there can generate code and translate between 60 different languages. Sometimes I only need the first part, sometimes the second. Two distinct models would be a lot smaller and run much faster (token-wise).

But they'd be stupider. The results for English and Python are much better because the model is also trained on Mandarin and Greek and Lisp even if you never make a request or receive a response in Mandarin, Greek or Lisp.

That's news to me since it's unlikely most of those weights are activated when responding to a coding prompt. Can you point to a source/paper that validates this claim?

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#339

Post by trollbridge »

reddguard wrote: post #93
trollbridge wrote: post #66

Earlier quoted context omitted.

Please don’t use Ollama. https://sleepingrobots.com/dreams/stop-using-ollama/

Doesn't Ollama use llama.cpp so their point stands even if they used it directly?

No, because Ollama is buggy. The first step to answering GPC’s question is to try using an up to date llama.cpp.
Post reply on HN