Gemini Diffusion
simonwillison.net
Gemini Diffusion
1–10 of 252 posts
Re: Gemini Diffusion
#2Interesting to see if GROQ hardware can run this diffusion architecture..it will be two time magnitude of currently known speed :O
Re: Gemini Diffusion
#3That's...ridiculously fast.
I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated.
Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. Encoding what isn't there seems like a hard problem, so even as models get smarter, they will continue to be handicapped by that lack of institutional knowledge, so to speak.
Imagine giving a large codebase to an incredibly talented developer and asking them to zero-shot a particular problem in one go, with only moments to read it and no opportunity to ask questions. More often than not, a less talented developer who is very familiar with that codebase will be able to add more value with the same amount of effort when tackling that same problem.
Re: Gemini Diffusion
#4I think the lede is being buried. This is a great and fast InstructGPT. This is absolutely going to be used in spell checks, codemods, and code editors.
Instant edits feature can surgically perform text edits fast without all the extra fluff or unsolicited enhancements.
I copied shadertoys, asked it to rename all variables to be more descriptive and pasted the result to see it still working. I'm impressed.
Re: Gemini Diffusion
#5Diffusion is more than just speed. Early benchmarks show it better at reasoning and planning pound for pound compared to AR.
This is because it can edit and doesn’t suffer from early token bias.
Re: Gemini Diffusion
#6Diffusion is more than just speed. Early benchmarks show it better at reasoning and planning pound for pound compared to AR. This is because it can edit and doesn’t suffer from early token bias.
This is a super interesting claim - can you point to these benchmarks?
Re: Gemini Diffusion
#7Nit: Diffusion isn't in place of transformers, it's in place of autoregression. Prior diffusion LLMs like Mercury [1] still use a transformer, but there's no causal masking, so the entire input is processed all at once and the output generation is obviously different. I very strongly suspect this is also using a transformer.
Re: Gemini Diffusion
#8Nit: Diffusion isn't in place of transformers, it's in place of autoregression. Prior diffusion LLMs like Mercury [1] still use a transformer, but there's no causal masking, so the entire input is processed all at once and the output generation is obviously different. I very strongly suspect this is also using a transformer. [1] https://www.inceptionlabs.ai/introducing-mercury
Interesting, my mind immediately went to block diffusion [0], but I think you are probably right.
Re: Gemini Diffusion
#9Nit: Diffusion isn't in place of transformers, it's in place of autoregression. Prior diffusion LLMs like Mercury [1] still use a transformer, but there's no causal masking, so the entire input is processed all at once and the output generation is obviously different. I very strongly suspect this is also using a transformer. [1] https://www.inceptionlabs.ai/introducing-mercury
Image diffusion models also use transformers nowadays. Here is the original "diffusion transformer" paper: https://arxiv.org/abs/2212.09748
Earlier image diffusion models used U-nets: https://en.wikipedia.org/wiki/U-Net
Re: Gemini Diffusion
#10Diffusion is more than just speed. Early benchmarks show it better at reasoning and planning pound for pound compared to AR. This is because it can edit and doesn’t suffer from early token bias.
A claim I believe (or want to) but can you point to any papers about this? I haven’t seen any papers at all or demos showing a revise diffusion text step. I’d reallly like to use one though.