Okay this tidbit is interesting to me "5–6 tok/s M2 -> 31–35 tok/s on an M5 Pro". So where will be in just another gen or two? my impression right now is that M5 gen is on the cusp of practicality for local inference. If techniques like OPs here, start to make the RAM situation more amenable, by the time we get to M6 or M7 (or AMD's equiv next gen APUs on TSMC N2 nodes), local AI could be ready to go much more mainst…
The speed of this perfectly correlates with the memory bandwidth of an M2 vs M5 Pro.
You’re right, Gemma isn’t the best model for coding (afaik more "everyday tasks" related). My first idea was to use Qwen, but its architecture was much more complex to implement in this stack. I chose Gemma so I wouldn’t spend all my time debugging custom kernels and could actually move the project forward with simpler approach
In defense of this model, Gemma is actually a very good general-purpose model that can work with multiple languages. I use it for spam classification and for processing dictation, which means that I hold the entire model in memory all of the time, which is somewhat problematic (64GB RAM total, but heavy usage by docker, databases, etc)
Gemma is a great reference model and it’s easy to work with. Once you have Gemma working well, then do the extra work to use Qwen as well.
I am using Gemma for a few tasks simply because it’s “good enough”.
Okay this tidbit is interesting to me "5–6 tok/s M2 -> 31–35 tok/s on an M5 Pro". So where will be in just another gen or two? my impression right now is that M5 gen is on the cusp of practicality for local inference. If techniques like OPs here, start to make the RAM situation more amenable, by the time we get to M6 or M7 (or AMD's equiv next gen APUs on TSMC N2 nodes), local AI could be ready to go much more mainst…
My assumption is that the difference is 90% from more memory. I'm making several assumptions because nothing here looks groundbreaking so I don't care to dig deeper, but the model + KV cache definitely cannot fit in memory on the 8GB machine, but probably can on the 24GB machine—or can at least get close. Assuming that this benchmark makes use of that, skipping SSD streaming will speed things up massively (I would have guessed much higher than the reported 6x speedup).
You’re a mad man - thank you! Do I understand correctly that Ollama doesnt do that, and that’s why responses hang forever on a M3 running the same model through Ollama?
The routed experts needed for the next tokens that are not already in memory need to be read from the SSD, so the speed becomes SSD reading bound and the larger the memory, the faster the inference.
Would be awesome if it ran Qwen (the MoE probably won't squeeze that low, but...). This because I have hardly been able to use Gemma for any sort of useful coding.
Gemma 4's tool calling was recently fixed; that was the main issue with agentic use in my experience.
Otherwise IMO it codes about as well as the Qwen MoE for PHP and SQL. It's a fully impressive model (though it is not as mindbendingly impressive as the 12B, which is outrageously good for its footprint)
I haven't tried it but it should work! You can try it and share your results, it would be really appreciated I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s Also, it must be easy to adjust for iPhones and iPads in theory