81–90 of 382 posts
Rushing to try it!
It’s like how Super Mario Bro was managed to put into 40KB of ROM
Claude was here.
There is one that could really improve the speed. Given almost all major models come with MTP head for speculative decoding. The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VRAM cache miss will be quite reduced.
If the technology demonstrates successful token rate improvement. future models could also come with pretraining heads to preload expert weights, and even make the training be aware of it.
I have a project that's almost ready to run DiffusionGemma as well. The two project might potentially work well together. I'm getting ~20tok/s on a 36GB M3 and there's strong possibility we might be able to crib faster kernels from each other. Feel free to reach out. (currently at https://github.com/mmastrac/diffgemma but not in a releasable state yet)
Cool project! I looked into it recently and thought that running diffusion models locally doesn't really make sense: https://eamag.me/2026/why-parallel-diffusion-llms-are-slow-o... What are your thoughts on this?
What I also learned is that MLX/vLLM is probably within ~20% or so of the absolute max perf on Mac. I found some improvements over what they were doing, but we're at the point where it's challenging to optimize without per-stepping kernels.
I found a few improvements over stock DiffusionGemma along the way, like using top-k attention, which drastically improves perf on my mac without sacrificing any of the benchmarks I was able to throw at it.
FWIW some of the issues with Gemma being slow on Mac are specific choices they've made in the architecture that make it challenging to make use various optimizations that have popped up recently. I think a Kimi K3-style network hybrid with the diffusion bits of DiffusionGemma could have some serious sway.
I think that diffusion still has an edge locally, but with some architecture tweaks and CPU improvements it would actually be a winner (ie: training the network for smaller token batch sizes or flexibility in attention heads, a less expensive attention mechanism, and others).
I think we strongly need something like that (shameless plug, I tried to build something around bitNet for the same reason: https://github.com/nickyreinert/bitNetRTR).
But at the end, all aproaches I saw, however genius they are: the actual results are always a mess. It's a better chat buddy, nothing else. It's e.g. far away from an decent coding assistants. I fine tuned Gemma with domain specific knowledge. Running it on a 16GB VRM GForce. Even then it's okai'sh but far way from a mind blowing experience. I ran some of the promised open source model on my 36GB MBPro M3, in Pi, Hermes, Continue. Can't compare the results to what Claude or Codex are offering.
You need at least something that's far away from consumer hardware, like those 7k'ish GForce machines with 96GB VRAM to get an idea of a good competitive model.
But... please, proof me wrong! =)
> The measured result is a reference point, not a performance ceiling. Claude was here.
Okay this tidbit is interesting to me "5–6 tok/s M2 -> 31–35 tok/s on an M5 Pro". So where will be in just another gen or two? my impression right now is that M5 gen is on the cusp of practicality for local inference. If techniques like OPs here, start to make the RAM situation more amenable, by the time we get to M6 or M7 (or AMD's equiv next gen APUs on TSMC N2 nodes), local AI could be ready to go much more mainst…
I have an M5 128GB. Being on the cusp of practical is a good description. It will run, but prefill and token gen are still slow relative to my consumer GPU box. It also gets very hot. If you’ve never heard the fans on Apple Silicon really spin up, it could surprise you. Makes the full GPU setup feel quiet by comparison. I think after the hardware market calms down the ticket is going to be a light laptop with a secon…