Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

271–280 of 383 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#271

Post by alexander2002 »

Keyframe wrote: post #168
giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

cant we have specialised models during onboarding i doubt a developer needs to know who king charles is?

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#272

Post by mandeepj »

spot5010 wrote: post #234
Keyframe wrote: post #168

Earlier quoted context omitted.

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

The human analogy would be that I don’t remember everything in the books I have read, but I do recall reading a particular book and can always look it up. So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.

> So, is there a way to train a neural network and then tune it forget a lot of the facts that can be easily retrieved, but keep the intelligence.

MoE (mixture of experts)?

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#273

Post by panarky »

glitchc wrote: post #255
giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

Indeed, it begs the question why we have "everything" models where instead we could have very efficient "something" models. Typical LLMs out there can generate code and translate between 60 different languages. Sometimes I only need the first part, sometimes the second. Two distinct models would be a lot smaller and run much faster (token-wise).

But they'd be stupider.

The results for English and Python are much better because the model is also trained on Mandarin and Greek and Lisp even if you never make a request or receive a response in Mandarin, Greek or Lisp.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#274

Post by antihipocrat »

Keyframe wrote: post #168
giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

Claude, create a webpage that provides family tree diagrams of all European royal families, clicking on each name expands the element to include any notable events from that person's life.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#276

Post by p0u4a »

kees99 wrote: post #138
nijave wrote: post #72

Earlier quoted context omitted.

Any idea if madvise helps? Admittedly I have very limited experience and only on Linux

MADV_SEQUENTIAL might help a bit, but not that much. Biggest problem here is throughput-vs-latency. With mmap()-ed file, for each pagefault, kernel will conservatively estimate block size to page in, so you'll have a ton of relatively small requests going to SSD. This would be IOPS-bound, and likely under-perform relative to maximum possible bytes/second throughput. With explicit read()/pread(), kernel & SSD can work…

> if software knows in advance which data chunk (expert) it'll need for the next token, it can load that in parallel with computing current token

You could actually use the model's MTP head to make a ~decent prediction on what experts would be activate in future tokens and preload them

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#277

Post by Keyframe »

alexander2002 wrote: post #271
Keyframe wrote: post #168

Earlier quoted context omitted.

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

cant we have specialised models during onboarding i doubt a developer needs to know who king charles is?

We can and we do https://en.wikipedia.org/wiki/Mixture_of_experts and even then - maybe not every developer needs python, maybe some do need C++ and King Charles..

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#278

Post by aitchnyu »

apitman wrote: post #192
micromacrofoot wrote: post #163

Earlier quoted context omitted.

The analysis has come back and the result is clear — the smoking gun is the belt-and-suspenders.

I got hit with my first belt-and-suspenders by Kimi K3 this morning. I normally use GPT. Is that a Claude-ism?

I got my first load-bearing in GLM 5.2 weeks back.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#279

Post by tedd4u »

quasarj wrote: post #260
xenonite wrote: post #48

With my M1 MBA, I am still on macOS 15. To compile it, just remove the two lines with opts.languageVersion = .version4_0 or surround them with if #available(macOS 26.0, *) { opts.languageVersion = .version4_0 } You'll miss out on a prefill speedup of 2.4x (as it yields 11.24x faster attention), according to the git comments, but it works. (On the 8-GPU-core MBA M1, I get 5-6 tok/s.)

Why are you still on 15?

I went back to 15 after accidentally upgrading to 26 because on MBA 13" screen the new UI design uses a lot of extra padding everywhere wasting screen space which is already at a premium (especially vertical). Hoping 27 fixes a lot of these issues.
Post reply on HN