Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

281–290 of 383 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#281

Post by tedd4u »

xenonite wrote: post #48

With my M1 MBA, I am still on macOS 15. To compile it, just remove the two lines with opts.languageVersion = .version4_0 or surround them with if #available(macOS 26.0, *) { opts.languageVersion = .version4_0 } You'll miss out on a prefill speedup of 2.4x (as it yields 11.24x faster attention), according to the git comments, but it works. (On the 8-GPU-core MBA M1, I get 5-6 tok/s.)

To build this (on macOS 15) I also had to do this (in Package.swift):

    @@ -4,8 +4,7 @@ import PackageDescription
     let package = Package(
         name: "TurboFieldfare",
         platforms: [
    -        .macOS(.v26),
    -        .iOS(.v26),
    +        .macOS(.v15)
         ],
         products: [
             .library(name: "TurboFieldfare", targets: ["TurboFieldfare"]),

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#282

Post by kstenerud »

Keyframe wrote: post #168
giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

I've been dealing with it on the post-training (runtime) side with a large codebase that contains many moving parts, many rules and requirements. Putting all of that into the context has already gone far past 1MB, so it's untenable.

Now I have all of that information stored in a pointers-to-resources fashion, where layers of small "directories" of triggers-to-information point the agent gradually towards deeper, more esoteric knowledge the more specific its needs become when tackling a goal.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#284

Post by AussieWog93 »

zkmon wrote: post #283

Please find a way to run Kimi K-3 on a 16 GB mac.

Apparently Kimi K3 has 104B parameters active at a time. So at 4 bits you'd need 52GB just to hold the active params.

That said, in theory this same technique should be able to run it on a 64GB Macbook, probably at <1 tps.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#285

Post by zozbot234 »

AussieWog93 wrote: post #284
zkmon wrote: post #283

Please find a way to run Kimi K-3 on a 16 GB mac.

Apparently Kimi K3 has 104B parameters active at a time. So at 4 bits you'd need 52GB just to hold the active params. That said, in theory this same technique should be able to run it on a 64GB Macbook, probably at <1 tps.

Only the sparse experts are 4-bit in native precision, and those take up ~25GB of active params. The dense parameters' native footprint is ~115GB. So in order to infer that natively on a 16GB machine you'd need to reload around ~135GB from disk at every token, which will take around 20.5 seconds at maximum 6.6 GB/s reading speed. This gives you a maximum theoretical performance of 176 tok/hr or 4224 tok/day when inferring at native precision. (Batching would be highly effective in aggregate since the bulk of what you're reloading is dense parameters, but your speed for any single session would still go down somewhat.) Of course all bets are off if you quantize the model highly; people are finding ways of fitting the whole thing in less than 600GB using extreme Q1 quants.

Mind you, the outlook for a 64GB RAM machine isn't that different. You'd get a faster SSD (around 2.2x performance) and be able to cache more of your dense params. So your performance would probably be around 4x compared to the 16GB case.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#286

Post by akmittal »

giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

Games used to render entire map in map in memory. Later they figured only the surroundings can we rendered. We can hopefully get same in AI

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#287

Post by musebox35 »

plandis wrote: post #167
swiftcoder wrote: post #149

Earlier quoted context omitted.

> I don't care who King Charles is every single time Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

Isn’t this essentially what MoE partially solves with varying levels of accuracy?

Sadly no. Despite the name, the experts are not routed per concept or topic but per token. So for the same sentence you might activate multiple experts for different tokens. What it solves is the distributed training and inference problem. As long as each expert fits a single gpu, coordinating the model evaluation is much easier and it is faster. It does not buy as much for running on a single device though still less costly than a dense version.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#288

Post by sznio »

Keyframe wrote: post #168
giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

distill. start with a general model, then create a smaller model trained only on relevant coding examples.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#289

Post by heyyeah »

rpdillon wrote: post #125
woadwarrior01 wrote: post #122

Earlier quoted context omitted.

I'm not making any moral judgements based on it, merely observing it.

The criticism is not that it's moralizing, but that it's boring.

100% agree. We're on Hacker News. We should be open to people who aren't native English speakers using LLMs to help them communicate their ideas and more importantly, cool projects without getting dragged for AI speak in the text.
Post reply on HN