Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

361–370 of 383 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#361

Post by panarky »

glitchc wrote: post #338
panarky wrote: post #273

Earlier quoted context omitted.

But they'd be stupider. The results for English and Python are much better because the model is also trained on Mandarin and Greek and Lisp even if you never make a request or receive a response in Mandarin, Greek or Lisp.

That's news to me since it's unlikely most of those weights are activated when responding to a coding prompt. Can you point to a source/paper that validates this claim?

It's well known and well documented in AI research.

Instead of taking syntactic shortcuts, the richer abstractions learned from multiple languages, and code, and math, and images and audio, ultimately make English comprehension and reasoning far stronger.

If you want citations, ask your favorite LLM how linguistic diversity prevents "surface memorization", overfitting on surface-level English patterns instead of representing the deeper concepts in latent vector space.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#363

Post by gitpusher42 »

febed wrote: post #362

Is the model response quality identical to the memory unconstrained model?

Yeah, it must be exactly the same. The same weights are used, nothing skipped or pruned. But it might have differ to MLX for greedy decode because of small floating-point nums difference

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#364

Post by gitpusher42 »

febed wrote: post #359
gitpusher42 wrote: post #301

Earlier quoted context omitted.

Yeah, the same ideas should work for qwen. You can try porting this engine to use Owen. Owen 3.6-35b-a3b was my initial idea, but I switched to Gemma because of its simpler architecture and kernels

Curious if the same idea could work with gpt-oss-120b? So one could run at least slowly on a Mac

Yeah, gpt-oss-120b is also MoE, so the same ssd-streaming and caching ideas should work. Feel free to fork and try implementing it!

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#365

Post by monegator »

Hi! Tried it and i'm impressed. The Mac app reports 4.4 token/s in the Mac Mini M2 with 8GB RAM. Not fast but still very much usable (my use rarely goes past from summarizing and generating pretty documentation). However, that mac sits in the rack cabinet and i ssh into it, so i would love to chat with it from the terminal, but because i generally use ollama i don't really know how to do that. Can someone help?

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#366

Post by dudefeliciano »

quasarj wrote: post #260
xenonite wrote: post #48

With my M1 MBA, I am still on macOS 15. To compile it, just remove the two lines with opts.languageVersion = .version4_0 or surround them with if #available(macOS 26.0, *) { opts.languageVersion = .version4_0 } You'll miss out on a prefill speedup of 2.4x (as it yields 11.24x faster attention), according to the git comments, but it works. (On the 8-GPU-core MBA M1, I get 5-6 tok/s.)

Why are you still on 15?

me too, everyone says it sucks and to wait for Golden Gate to release.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#367

Post by dofm »

febed wrote: post #360
dofm wrote: post #69

Earlier quoted context omitted.

Gemma 4's tool calling was recently fixed; that was the main issue with agentic use in my experience. Otherwise IMO it codes about as well as the Qwen MoE for PHP and SQL. It's a fully impressive model (though it is not as mindbendingly impressive as the 12B, which is outrageously good for its footprint)

Which model do you think currently hits the highest size-to-performance ratio for agentic tasks like tool calling?

OK so I don't have enormous agentic coding experience yet (I'm still learning about the tech more than coding with it) but:

- The consensus is that if you have a properly kitted out PC with enough VRAM, the Qwen 3.6 27B dense model is the one to beat.

Bit slow on my M1 Max so I haven't bothered with it much but I have no reason to doubt the consensus. Prism ML's new Ternary Bonsai variant of it makes it much easier to play with this model in limited RAM, but in my own toy experiments I have seen Ternary Bonsai get very stuck in thinking loops. There is another post-train variant from BottleCap called ThinkingCap, which you could try.

- The Qwen 35B MoE model is really impressive for code generation. I personally would pick this one for an older Mac or a machine with smaller VRAM; it's pretty fast, has good built-in MTP, great tool-calling.

- The Gemma 4 26B MoE has a similar capability but biased more to writing than coding. I think it writes really well, and it seems to have good knowledge of e.g. WordPress coding, SQL etc. Tool-calling let it down for agentic coding, and I haven't retested it since they fixed that

- The dense 31B Gemma 4 is large and runs slowly on my machine, but has very good general knowledge, writes well, so it should I think be better than the Qwen 27B for research tasks, and it should now be pretty solid at tool-calling.

- If you don't have much VRAM, you are not doing much coding (e.g. you want short snippets) and you want to experiment with local LLMs and perhaps in particular image analysis, the Gemma 4 12B is fun. It has an integrated vision decoder which is very impressive. Codewise, it's going to fail on long context tasks.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#368

Post by febed »

dofm wrote: post #367
febed wrote: post #360

Earlier quoted context omitted.

Which model do you think currently hits the highest size-to-performance ratio for agentic tasks like tool calling?

OK so I don't have enormous agentic coding experience yet (I'm still learning about the tech more than coding with it) but: - The consensus is that if you have a properly kitted out PC with enough VRAM, the Qwen 3.6 27B dense model is the one to beat. Bit slow on my M1 Max so I haven't bothered with it much but I have no reason to doubt the consensus. Prism ML's new Ternary Bonsai variant of it makes it much easier t…

Thanks a lot for the recommendations!

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#369

Post by dofm »

febed wrote: post #368
dofm wrote: post #367

Earlier quoted context omitted.

OK so I don't have enormous agentic coding experience yet (I'm still learning about the tech more than coding with it) but: - The consensus is that if you have a properly kitted out PC with enough VRAM, the Qwen 3.6 27B dense model is the one to beat. Bit slow on my M1 Max so I haven't bothered with it much but I have no reason to doubt the consensus. Prism ML's new Ternary Bonsai variant of it makes it much easier t…

Thanks a lot for the recommendations!

No worries. YMMV for coding things but I've learned a lot more about LLMs this way than I think I could have from just reading about and using cloud LLMs.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#370

Post by ycui1986 »

IsTom wrote: post #115
hacklas wrote: post #94

Earlier quoted context omitted.

Worth mentioning why this is harder than it looks. There is a different set of experts at every layer, and each layer has a small router that decides which ones to use. The router needs to look at the state produced by the experts below it. Drafted tokens from the MTP head can be used to predict which experts the first layer will want, but not beyond that. To know what layer 10 experts needs, you have to run layers 1…

That kind of sounds like a branch predictor in a CPU.

exactly. we need branch predictor for expert weight.
Post reply on HN