Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DiffusionGemma: 4x Faster Text Generation

blog.google

Post reply on HN
Print view View original

61–70 of 94 posts

Re: DiffusionGemma: 4x Faster Text Generation

#61

Post by elxr »

bpavuk wrote: post #43
vineyardmike wrote: post #9

Recently I had switched to OpenCode to try out many of the Non-US-Frontier-Labs models. My unexpected favorite model to use was Mercury (a diffusion model). Not because it was “smart” but because it was stupid fast. It was more of a pair-programming experience instead of the SOTA agentic experience of prompting and waiting. Honestly, it was also way more fun and brought back some of the pre-AI coding experience while…

YESSSS!!! speed is THE way! I like my boilerplate POJOs/data classes generated at breakneck pace of 300+ tok/s, Flash-Lite is more useful than GPT-5.5 for me this way. if it's too slow, you just stay in that goddamn async death loop

For boilerplate, yeah. But when asking research or exploratory questions, or weighing whether a feature is well designed, or asking "can I implement _x_ feature using these libraries without introducing unnecessary complexity", then GPT-5.5 medium is still fast enough.

10-20 seconds times a couple turns on a new feature isn't bad. Kimi is also similarly fast if not faster.

I do agree with smaller models for more constrained/routine tasks though.

Re: DiffusionGemma: 4x Faster Text Generation

#62

Post by irthomasthomas »

vineyardmike wrote: post #9

Recently I had switched to OpenCode to try out many of the Non-US-Frontier-Labs models. My unexpected favorite model to use was Mercury (a diffusion model). Not because it was “smart” but because it was stupid fast. It was more of a pair-programming experience instead of the SOTA agentic experience of prompting and waiting. Honestly, it was also way more fun and brought back some of the pre-AI coding experience while…

Mercury-2 is amazing. I am using it frequently as the arbiter in llm-consortium The context window is relatively small, so to make it work with larger consortiums I can construct a recursive sort-of meta consortium like this:

  llm consortium save cns-glm -m glm-5.2 -n 5 --arbiter mercury-2 --judging-method rank

  llm consortium save cns-kimi -m k2.6 -n 5 --arbiter mercury-2 --judging-method rank

  llm consortium save cns-meta-glm-kimi -m cns-glm -m cns-kimi --arbiter mercury-2 --judging-method synthesis
Now when I prompt cns-meta-glm-kimi it will pick the best of five from kimi and glm before creating a synthesis from the two winners.

Re: DiffusionGemma: 4x Faster Text Generation

#64

Post by dr_kiszonka »

simonw wrote: post #30

NVIDIA are hosting a free endpoint for this one, details at https://build.nvidia.com/google/diffusiongemma-26b-a4b-it - you have to create an account and (I think) verify a phone number too. (I got it to draw a pelican: https://tools.simonwillison.net/markdown-svg-renderer#url=ht... )

Maybe with very fast models you could request animation frames, e.g., frame 1) right foot at 12, left foot at 6; frame 2) right foot at 3, left foot at 9, etc.?

And instead of reporting tps, you would - of course! - report pfps (pelican frames per second).

Re: DiffusionGemma: 4x Faster Text Generation

#65

Post by SwellJoe »

irthomasthomas wrote: post #62
vineyardmike wrote: post #9

Recently I had switched to OpenCode to try out many of the Non-US-Frontier-Labs models. My unexpected favorite model to use was Mercury (a diffusion model). Not because it was “smart” but because it was stupid fast. It was more of a pair-programming experience instead of the SOTA agentic experience of prompting and waiting. Honestly, it was also way more fun and brought back some of the pre-AI coding experience while…

Mercury-2 is amazing. I am using it frequently as the arbiter in llm-consortium The context window is relatively small, so to make it work with larger consortiums I can construct a recursive sort-of meta consortium like this: llm consortium save cns-glm -m glm-5.2 -n 5 --arbiter mercury-2 --judging-method rank llm consortium save cns-kimi -m k2.6 -n 5 --arbiter mercury-2 --judging-method rank llm consortium save cns-…

I've found the average output of many suboptimal models is still suboptimal, especially when it comes to judging the accuracy/correctness of the work of other models.

I did some benchmarks recently of how well various models find security vulnerabilities, and then follow up testing of the judging process of whether the models found the right bug and whether other bugs it reported were false positives or legitimate other bugs. A committee of good-not-great models (DeepSeek, MiMo, Gemma 4) cannot replicate the accuracy of Opus by itself. Even when all three of the other models disagreed with Opus, Opus was almost always the one that was actually right.

It's an interesting area for research. And, a model that's very fast can make a lot more attempts at a solution, and in cases where there is an unambiguous "right" solution that can be proven by some sort of static rule, "very fast" may be a useful characteristic. Small classification problems, where you need to make thousands of decisions about some specific aspect of a large corpus of data, seems like a sweet spot for a model like Mercury.

Re: DiffusionGemma: 4x Faster Text Generation

#66

Post by bArray »

SwellJoe wrote: post #60

Google keeps flexin'. It's surprising that Gemini isn't more competitive against Claude or OpenAI models for code and agentic use, because it's clear Google still has some of the best AI people in the business. But, I guess Google is focused on stuff that runs on phones and near-realtime use cases, rather than the big thinky LLMs. All these efficiency improvements seem likely to be really important to the future of A…

I think Google will win out in the end. They are concentrating on what matters, performance per watt, and performance per dollar. They are building their own inference hardware and are working towards edge-computing which removes latency and compute overheads. These big LLMs are not yet cost effective, Google is just letting them burn their investment funds to "sell" to consumers at below cost.

After the AI bubble bursts, it will be the likes of Google that come out the other side still wearing their shirts. I think this bubble is out to scalp some giants.

Re: DiffusionGemma: 4x Faster Text Generation

#67

Post by zozbot234 »

SwellJoe wrote: post #60

Google keeps flexin'. It's surprising that Gemini isn't more competitive against Claude or OpenAI models for code and agentic use, because it's clear Google still has some of the best AI people in the business. But, I guess Google is focused on stuff that runs on phones and near-realtime use cases, rather than the big thinky LLMs. All these efficiency improvements seem likely to be really important to the future of A…

Fable's costs are twice Opus' and it's clearly quite competitive with GPT-Pro, so that seems like it might be a good option for you if the trigger-happy safeguards aren't too much of a problem. Google has their own "Deep Research" option in this space which seems to work well.

The nice thing about DeepSeek is its ability to be run on local hardware, with no API costs involved. If you care deeply about that, then it being a bit worse than Opus or GPT isn't really a problem.

Re: DiffusionGemma: 4x Faster Text Generation

#68

Post by aleksiy123 »

onlyrealcuzzo wrote: post #49
bee_rider wrote: post #46

Earlier quoted context omitted.

How do you measure “true” complexity? Cyclomatic seems a bit… I dunno, artificial? Blunt? But it has the benefit of being defined.

There's a ton of research on this in the 80s... and interestingly, I haven't seen a lot of recent research. Surprisingly, it seems most languages don't have a standard package to do a lot of these detections. Ruby has Flay to detect similarity (something LLMs are prone to do). Basically re-write a huge function with only a couple of minor differences that should probably be params... One of the things I rely on most…

Interesting, I do think blending the fuzziness of models with the determinism of hard checks/conformance is the way too go.

But using some kind of metrics as guardrails/steering seems interesting.

Re: DiffusionGemma: 4x Faster Text Generation

#69

Post by yorwba »

LarsDu88 wrote: post #53

Does anyone know of the current intrinsic limitations with Diffusion text models compared to autoregressive? I ran this question by ChatGPT and Claude and they came up with limitations in GRPO RLVR, but I'm not sure..

The intrinsic limitation of text diffusion is that natural text contains serial dependencies where a word at the beginning of the text strongly influences what comes later, and if there is a long enough dependency chain within a diffusion block, the small number of diffusion steps may not be enough to resolve all dependencies, so that you end up with incoherent output.

Re: DiffusionGemma: 4x Faster Text Generation

#70

Post by irthomasthomas »

SwellJoe wrote: post #65
irthomasthomas wrote: post #62

Earlier quoted context omitted.

Mercury-2 is amazing. I am using it frequently as the arbiter in llm-consortium The context window is relatively small, so to make it work with larger consortiums I can construct a recursive sort-of meta consortium like this: llm consortium save cns-glm -m glm-5.2 -n 5 --arbiter mercury-2 --judging-method rank llm consortium save cns-kimi -m k2.6 -n 5 --arbiter mercury-2 --judging-method rank llm consortium save cns-…

I've found the average output of many suboptimal models is still suboptimal, especially when it comes to judging the accuracy/correctness of the work of other models. I did some benchmarks recently of how well various models find security vulnerabilities, and then follow up testing of the judging process of whether the models found the right bug and whether other bugs it reported were false positives or legitimate ot…

I have had a better experience with my own use. I use it every day and it rarely fails to improve tasks. Perhaps the prompts and rubrics make a difference. And finding bugs is one of the better use cases because it is essentially a search problem. As long as models are non-deterministic and there is some diversity in training data, then an ensemble that iterates on the problem is more likely to cover the ground needed to find solve a problem.

Some tasks benefit from this approach more than others. There was a paper from google on a version they made which was very similar and achieved SOTA then on planning and pathfinding benchmarks.

edit:

Mind Evolution paper https://deepmind.google/research/publications/122391/

(That was a month after I published llm-consortium :) https://xcancel.com/karpathy/status/1870692546969735361

Post reply on HN