I always thought that fundamentally diffusors were the cooler idea of the two
DiffusionGemma: 4x Faster Text Generation
71–80 of 94 posts
Re: DiffusionGemma: 4x Faster Text Generation
#72Does anyone know of the current intrinsic limitations with Diffusion text models compared to autoregressive? I ran this question by ChatGPT and Claude and they came up with limitations in GRPO RLVR, but I'm not sure..
CoT legibility largely disappears which is quite concerning from a safety perspective
Re: DiffusionGemma: 4x Faster Text Generation
#73Does anyone know of the current intrinsic limitations with Diffusion text models compared to autoregressive? I ran this question by ChatGPT and Claude and they came up with limitations in GRPO RLVR, but I'm not sure..
The intrinsic limitation of text diffusion is that natural text contains serial dependencies where a word at the beginning of the text strongly influences what comes later, and if there is a long enough dependency chain within a diffusion block, the small number of diffusion steps may not be enough to resolve all dependencies, so that you end up with incoherent output.
The obvious solution is to simply do more steps for larger sequences though, right?
How exactly does this work with CoT?
Re: DiffusionGemma: 4x Faster Text Generation
#74Some of these comments miss the advantage of diffusion. This is will have a big impact on edge devices, such as your phone or the GPU in your computer. An LLM's decoder computes tokens one-at-a-time because attention has to account for each previous token. The existing LLM decoders scale well when you have enough load to batch many inferences together. Diffusion of limited benefit there. On edge you have a different…
You’re mostly right but conflating attention with autoregressive/causal which is the real issue that prevents you from using more compute You can use diffusion with attention, and this model does in fact use attention
Yes, I should have said autoregressive.
Re: DiffusionGemma: 4x Faster Text Generation
#75[flagged]
Re: DiffusionGemma: 4x Faster Text Generation
#76Google keeps flexin'. It's surprising that Gemini isn't more competitive against Claude or OpenAI models for code and agentic use, because it's clear Google still has some of the best AI people in the business. But, I guess Google is focused on stuff that runs on phones and near-realtime use cases, rather than the big thinky LLMs. All these efficiency improvements seem likely to be really important to the future of A…
Google is clearly gimping the gemma models. There is a 122b gemma 4 that was never released, but was a part of the announcement tweet. Plus they weren't going to release MTP until people figured out they're running it on the pixels
Re: DiffusionGemma: 4x Faster Text Generation
#77NVIDIA are hosting a free endpoint for this one, details at https://build.nvidia.com/google/diffusiongemma-26b-a4b-it - you have to create an account and (I think) verify a phone number too. (I got it to draw a pelican: https://tools.simonwillison.net/markdown-svg-renderer#url=ht... )
Thought of you this afternoon, "after you click the record button can you make a 'boop, boop, boop, clack!' like a lead-in from a from a clapboard (using web audio synthesis apis)?"
...was quite surprising the result!
Re: DiffusionGemma: 4x Faster Text Generation
#78Do diffusion models support tool calls? If so is the tool call support on par with autoregressive models or worse? (edited spelling)
omlx.server - WARNING - POST /v1/chat/completions -> 400: Tool calling is not supported with diffusion models.
Re: DiffusionGemma: 4x Faster Text Generation
#79I wish labs would do QAT and release these quants, at this point looking at releases of bf16 without QAT feels like looking at half backed bread, we can quantized it but it is not the same as QAT. Or I am missing something here ?
Re: DiffusionGemma: 4x Faster Text Generation
#80Google keeps flexin'. It's surprising that Gemini isn't more competitive against Claude or OpenAI models for code and agentic use, because it's clear Google still has some of the best AI people in the business. But, I guess Google is focused on stuff that runs on phones and near-realtime use cases, rather than the big thinky LLMs. All these efficiency improvements seem likely to be really important to the future of A…
Google is clearly gimping the gemma models. There is a 122b gemma 4 that was never released, but was a part of the announcement tweet. Plus they weren't going to release MTP until people figured out they're running it on the pixels
I dunno about that. Gemma 4 is probably the best model for general self-hosted use for almost everyone that doesn't have a data center in their basement. They didn't have to release it at all, and they didn't have to release speculative decoding drafters, and they didn't have to release the QAT version of the models that makes the 4-bit quantization perform very close to the bigger versions, and can run in 32GB. I'd love a 122B version of it, and I didn't realize they'd ever announced one was coming (though I remember there being speculation about it). But, also, I'm happy they're doing so much with it. They've got all the sizes covered, it has great prose for an LLM, better prose than even most larger models, it's got great audio and vision, and broad language support. As self-hosted general purpose models go, it's the total package.
Qwen 3.6 is maybe better for code (though I'm beginning to think otherwise after some benchmarking I've been doing, where Gemma 4 has been overperforming expectations), but for just about anything else, Gemma 4 is the one.
If they're gimping it, why is nobody else making a better one that small?