[dead]
DiffusionGemma: 4x Faster Text Generation
91–94 of 94 posts
Re: DiffusionGemma: 4x Faster Text Generation
#92Thanks gemma team for this release.
Compared to autoregressive decoding, diffusion is huge for local MoE inference because of the improved token generation efficiency, especially for normal GPU + ram offload setting.
However, there are models which are better positioned on the performance vs memory pareto front, i.e. dense models, so I'll just wait.
P.S. QAT is really something as it reduces the performance fluctuations compared to the normal one. Thanks again.
Re: DiffusionGemma: 4x Faster Text Generation
#93Recently I had switched to OpenCode to try out many of the Non-US-Frontier-Labs models. My unexpected favorite model to use was Mercury (a diffusion model). Not because it was “smart” but because it was stupid fast. It was more of a pair-programming experience instead of the SOTA agentic experience of prompting and waiting. Honestly, it was also way more fun and brought back some of the pre-AI coding experience while…
If you can run your tests fast and cheaply, and have metrics that show what bad/sloppy code is that are cheap & fast to generate, a worse fast model can outperform a far better far slower model if you value time... I've had pretty good success with LLMs after putting in place metrics to measure true complexity (not cyclomatic), and automatically pushing back everything until the added complexity is within reason for…
I wonder if a dedicated client or mode in a client would provide some benefits. Might also be interesting to do adversarial stuff too where it argues with itself or another model
Re: DiffusionGemma: 4x Faster Text Generation
#94Do diffusion models support tool calls? If so is the tool call support on par with autoregressive models or worse? (edited spelling)
omlx.server - WARNING - POST /v1/chat/completions -> 400: Tool calling is not supported with diffusion models.
Pull request #1837 that enables tool calls on supported diffusion models was merged as 7c1971e today. I previously tested mlx-community/diffusiongemma-26B-A4B-it-8bit on a custom patched version of omlx in the Zed Agent Panel. The majority of the tool calls worked.
What didn't work reliably was specifically write tool calls and this is not resolved by the pull request. But as far as I understand the problem is not the inference framework but the root issue is that DiffusionGemma emits incorrect JSON.
When `content` contains `, ` inside a string value, the decoder splits there and emits the remainder as a nonsensical JSON key. So `{"path": "f.py", "content": "def f(x, y):\n return x"}` becomes `{"path": "f.py", "content": "def f(x", "y):\n return x": ...}`.
I wondered if the JSON issue might be related to quantization and tested the BF16 variant of google/diffusiongemma-26b-a4b-it via NVIDIA NIM. The model did not show the delimiter-splitting bug. It did however have a quote-handling issue. Among others it duplicated tripple quotes (`"""..."""` becomes `""""""...""""""`).