Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini Diffusion

simonwillison.net

Post reply on HN
Print view View original

41–50 of 252 posts

Re: Gemini Diffusion

#41

Post by boroboro4 »

quantadev wrote: post #26

Anyone able to summarize the current 'hold up' with diffusion models? I know exactly how Transformers work, but I'm not a diffusion expert. Diffusion is so much more powerful tho (from what I know) it seems like diffusion would already be beating Transformers. Why isn't it?

Diffusion is about what goes into the model and what’s a result (in this case it’s denoising of the content) as opposed to autoregressive models (where the process is to predict continuation based on prefix). It’s orthogonal to model architecture, which can be transformer or (for example) mamba. I’m pretty sure Gemini diffusion is transformer too.

Diffusion brings different set of trade offs, and as you can see it improves speed but I would expect it increases compute required for generation. But this is hard to say for sure without knowing their exact sampling process.

Interestingly we have opposite direction in case with gpt-4o, OpenAI made autoregressive image generation model and it seems it works great.

Re: Gemini Diffusion

#42

Post by dleeftink »

KingMob wrote: post #40
heliophobicdude wrote: post #4

I think the lede is being buried. This is a great and fast InstructGPT. This is absolutely going to be used in spell checks, codemods, and code editors. Instant edits feature can surgically perform text edits fast without all the extra fluff or unsolicited enhancements. I copied shadertoys, asked it to rename all variables to be more descriptive and pasted the result to see it still working. I'm impressed.

Spell check? Isn't that a well-solved problem at this point?

Solved how? Language is always evolving

Re: Gemini Diffusion

#43

Post by mountainriver »

martincsweiss wrote: post #6
mountainriver wrote: post #5

Diffusion is more than just speed. Early benchmarks show it better at reasoning and planning pound for pound compared to AR. This is because it can edit and doesn’t suffer from early token bias.

This is a super interesting claim - can you point to these benchmarks?

https://github.com/HKUNLP/diffusion-vs-ar

Re: Gemini Diffusion

#44

Post by mdp2021 »

hansvm wrote: post #34
mountainriver wrote: post #5

Diffusion is more than just speed. Early benchmarks show it better at reasoning and planning pound for pound compared to AR. This is because it can edit and doesn’t suffer from early token bias.

AR doesn't inhibit long planning processes, but some popular, modern instantiations of AR have that flaw. AR in general is critical for learning the right distribution.

> AR in general is critical for learning the right distribution

Could you please clarify that?

Re: Gemini Diffusion

#45

Post by mdp2021 »

mountainriver wrote: post #43
martincsweiss wrote: post #6

Earlier quoted context omitted.

This is a super interesting claim - can you point to these benchmarks?

https://github.com/HKUNLP/diffusion-vs-ar

I.e.: https://arxiv.org/html/2410.14157v3

# Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning

Re: Gemini Diffusion

#46

Post by 8n4vidtmkvmk »

airstrike wrote: post #3

That's...ridiculously fast. I still feel like the best uses of models we've seen to date is for brand new code and quick prototyping. I'm less convinced of the strength of their capabilities for improving on large preexisting content over which someone has repeatedly iterated. Part of that is because, by definition, models cannot know what is not in a codebase and there is meaningful signal in that negative space. En…

That's not been my experience so far. LLMs are good at mimicking existing good, it doesn't usually bring in new things when not asked. Sometimes I have to go out of my way to point to other bits of code in the project to copy from because it hasn't ingested enough of the codebase.

That said, a negative prompt like we have in stable diffusion would still be very cool.

Re: Gemini Diffusion

#47

Post by 8n4vidtmkvmk »

KingMob wrote: post #40
heliophobicdude wrote: post #4

I think the lede is being buried. This is a great and fast InstructGPT. This is absolutely going to be used in spell checks, codemods, and code editors. Instant edits feature can surgically perform text edits fast without all the extra fluff or unsolicited enhancements. I copied shadertoys, asked it to rename all variables to be more descriptive and pasted the result to see it still working. I'm impressed.

Spell check? Isn't that a well-solved problem at this point?

How does grammarly exist then? Must be some secret sauce in there.

Re: Gemini Diffusion

#48

Post by mattnewton »

crystal_revenge wrote: post #32
synapsomorphy wrote: post #7

Nit: Diffusion isn't in place of transformers, it's in place of autoregression. Prior diffusion LLMs like Mercury [1] still use a transformer, but there's no causal masking, so the entire input is processed all at once and the output generation is obviously different. I very strongly suspect this is also using a transformer. [1] https://www.inceptionlabs.ai/introducing-mercury

> so the entire input is processed all at once I’m a bit confused by this statement. Autoregresive LLMs also process the entire input “at once” otherwise tricks like speculative decoding wouldn’t work. Can you clarify what you mean by this?

tokens in a diffusion model typically look like encoders where the tokens earlier in the sentence can “see” tokens later in the sentence, attending to their values. Noise is iteratively removed from an entire buffer all at once in a couple steps.

Versus one step per token, where autoregressive models only attend to previous tokens.

Re: Gemini Diffusion

#50

Post by shreezus »

Is anyone else totally blown away by this? I feel like it’s easily the biggest announcement out of IO, however it’s been overshadowed by Veo 3 etc.

Diffusion models for code generation are a big deal. If they are using transformers this would likely fall into the DiT bucket (diffusion transformers). I had previously worked on use cases that leveraged U-Net diffusion several years ago and there was quite a bit of interest in hybrid models. I expect to see further leaps in the diffusion space in the near future.

Post reply on HN