How does this compare to DwarfStar4?
DS4 is designed to do real-work. Gemma 4 is not going to cut it.
[dead]
Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
91–100 of 382 posts
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#92Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory.
Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and practicality, they just leave it to whoever sets up infrastructure to worry about. I wouldn't be surprised if frontier AI could be drastically cheaper if they just finetune and optimize their models to not consume all available RAM to only access less than 10% of the models knowledge.
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#93You’re a mad man - thank you! Do I understand correctly that Ollama doesnt do that, and that’s why responses hang forever on a M3 running the same model through Ollama?
Please don’t use Ollama. https://sleepingrobots.com/dreams/stop-using-ollama/
Doesn't Ollama use llama.cpp so their point stands even if they used it directly?
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#94There are a lot of SSD streaming engines these days. But few to actually try some hard features. There is one that could really improve the speed. Given almost all major models come with MTP head for speculative decoding. The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VR…
Worth mentioning why this is harder than it looks.
There is a different set of experts at every layer, and each layer has a small router that decides which ones to use.
The router needs to look at the state produced by the experts below it.
Drafted tokens from the MTP head can be used to predict which experts the first layer will want, but not beyond that. To know what layer 10 experts needs, you have to run layers 1-9 which means loading their experts.
So, yes, instead of a next-token drafter like MTP, you'd want something trained to predict the expert activation across all layers at once.
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#95I have a project that's almost ready to run DiffusionGemma as well. The two project might potentially work well together. I'm getting ~20tok/s on a 36GB M3 and there's strong possibility we might be able to crib faster kernels from each other. Feel free to reach out. (currently at https://github.com/mmastrac/diffgemma but not in a releasable state yet)
It is super cool! Diffusion Gemma was released around the middle of my project, and I seriously considered switching to it. But I decided to finish the project as it was. I believe it would be a perfect match! Feel free to use any parts of my project or drop me a message. There’s my LinkedIn link at the end of the readme. Or I will drop you a message later!
Awesome. I may need to finally bite the bullet and upgrade my macOS to test out the MPP approach you've taken.
I've got a number of tiled-load kernels, and a top-k attention kernel that you might find interesting.
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#96Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…
[deleted]
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#97Dont want to crash the party here, but I am still sceptic about all those on-premise-llm-approaches. I think we strongly need something like that (shameless plug, I tried to build something around bitNet for the same reason: https://github.com/nickyreinert/bitNetRTR ). But at the end, all aproaches I saw, however genius they are: the actual results are always a mess. It's a better chat buddy, nothing else. It's e.g.…
This is an odd comment: the project is right there for you to use, so just try it and see if it holds up to the claims? Then you can comment about the fact that it either doesn't hold up, with numbers to back that up, or on how awesome it is because it works =)
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#98This is really neat! Question, on that MacBook Pro with presumably more RAM, is it still holding itself back in the RAM department?
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#99> The measured result is a reference point, not a performance ceiling. Claude was here.
"I ran more than 100 experiments. Most didn’t work. A few got me here."
And here.
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#100> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro. Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
I am relying quite heavily on system caching and pread. And yeah, M5 is a way faster and I can guess Mac can cache something, even if process stays under 2gb.
It was 83ms read per token for M2 and 12ms on M5 pro. Total is 163ms/tok vs 30ms/tok for M5. So yeah, there is a faster read and faster gpu processing