Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini Diffusion

simonwillison.net

Post reply on HN
Print view View original

221–230 of 252 posts

Re: Gemini Diffusion

#221

Post by airstrike »

lukan wrote: post #219
skydhash wrote: post #213

Earlier quoted context omitted.

Yes simplicity is always important, but it does not equate easiness. The axe of simple to complex is independent of the axe of easy to hard. It may be easy to apply patterns blindly to your codebass and make it complex. Just how it is easy to write naive and simple code that then becomes difficult to work with. The mark of a good programmer is to balance all of these so that it’s easy to work with the codebase on an…

"At each stage, you get enough feedback to judge the right direction for the next iteration." Depends on the project I would say. What do you do, if all of a sudden the requirements change again? Or the plattform evolved/degraded? Then you compromise - and I can better compromise with simple solution. And I would never claim simple equals easy. Rather the opposite. Like you said, it is easy to make complex things. Al…

Re: this whole conversation, you might find this quick video a worthwhile watch https://www.youtube.com/watch?v=wrwxC9taL8w

Re: Gemini Diffusion

#223

Post by thomasahle »

> Now, diffusion LMs take this idea further. BERT can recover 15% of masked tokens ("noise"), but why stop here. Let's train a model to recover texts with 30%, 50%, 90%, 100% of masked tokens.

> Once you've trained that, in order to generate something from scratch, you start by feeding the model all [MASK]s. It will generate you mostly gibberish, but you can take some tokens (let's say, 10%) at random positions and assume that these tokens are generated ("final")

This is clearly wrong. If you actually froze 10% of gibberish tokens, your output would be terrible!

What you actually do in discrete statespace diffusion (see e.g. [1]) is to allow every token to change at every time step.

You combine this with a "schedule" that allows the model to know how close it is to being done. E.g. at t=0/20 the changes will be large, and at t=19/20 only small refinements are made.

Update: There is actually a kind of model that "greedily" freezes the top-p most confident tokens, similar to what the blog post describes (though not at random!) this is called MaskGit [2], but it is not a diffusion model and doesn't work as well.

Btw, you can also just use "continuous diffusion" with a transformer/bert model, where you've removed the top softmax layer. Then everything works as normal with Gaussian noise, and you just do softmax at the the final time step.

[1] https://arxiv.org/abs/2107.03006

[2] https://arxiv.org/abs/2202.04200

Re: Gemini Diffusion

#224

Post by x_may »

WhitneyLand wrote: post #222

It’s promising but they need to find a way to add reasoning techniques to DLLMs. Were a ways out still from matching from frontier performance.

Obviously its not at the scale of the top auto-regressive models yet but there are some OSS models https://github.com/dllm-reasoning/d1

Re: Gemini Diffusion

#225

Post by lcnPylGDnU4H9OF »

beernet wrote: post #175
ealexhudson wrote: post #149

Earlier quoted context omitted.

Perhaps your content quality meter needs a recalibration?

How so? What makes this blog stand out in terms of quality? I prefer a constructive discussion over personal questions, maybe you should, too.

It's the name at the top. This particular author has been active with LLM posts at least since the popularity explosion of ChatGPT and all of their posts on that topic seem to be well-informed (and they are otherwise community-famous for co-authoring Django). To your point, the content is only as special as the author's reputation makes it, which will be different from reader to reader.

https://en.wikipedia.org/wiki/Simon_Willison

Re: Gemini Diffusion

#226

Post by fragmede »

Incipient wrote: post #111
fragmede wrote: post #75

Earlier quoted context omitted.

Which LLM are you using? what LLM tool are you using? What's your tech stack that you're generating code for? Without sharing anything you can't, what prompts are you using?

Was more of a general comment - I'm surprised there is significant variation between any of the frontier models? However, vscode with various python frameworks/libraries; dash, fastapi, pandas, etc. Typically passing the 4-5 relevant files in as context. Developing via docker so I haven't found a nice way for agents to work.

> I'm surprised there is significant variation between any of the frontier models?

This comment of mine is a bit dated, but even the same model can have significant variation if you change the prompt by just a few words.

https://news.ycombinator.com/item?id=42506554

Re: Gemini Diffusion

#227

Post by boroboro4 »

atq2119 wrote: post #57
boroboro4 wrote: post #41

Earlier quoted context omitted.

Diffusion is about what goes into the model and what’s a result (in this case it’s denoising of the content) as opposed to autoregressive models (where the process is to predict continuation based on prefix). It’s orthogonal to model architecture, which can be transformer or (for example) mamba. I’m pretty sure Gemini diffusion is transformer too. Diffusion brings different set of trade offs, and as you can see it im…

Diffusion could potentially be more efficient for local inference. With auto-regressive models, token generation is basically one token at a time, and so is not compute intensive at all -- it's bandwidth bound. With diffusion, you always run the model on a decently sized batch of tokens, so you should be (close to) compute bound even for local inference. If the "output quality per compute" is roughly the same for dif…

Yeah, it might be a win for local inference.

I think "output quality per compute" will be loss for diffusion models, but it might be similar (or even better?) for "output quality per number of parameters". Which will still make it better for local inference.

However autoregressive models also have own way of dealing with low compute utilization - it's speculative decoding. You can use smaller (and faster) model to generate bunch of different possible continuations and verify all of them at once. I think Eagle3 for example achieves ~8 tokens per iteration speedup this way (and to be frank I believe it can be even better).

Re: Gemini Diffusion

#228

Post by cztomsik »

slickytail wrote: post #126
cztomsik wrote: post #100

I have no idea how it works actually (in google) but I wouldn't be surprised if it was just post-training because recently RWKV people did something similar: They replaced the whole attention mechanism with WKV (forward-only linear attention), and created such franken-stein just by post-training. The big wow moment about that is that it sort of implies that most of the useful knowledge is in the FFN, and attention it…

The relative unimportance of the exact SDPA attention in use in modern transformers is already known: https://arxiv.org/abs/2111.11418 The FFN, normalization, and residual connections are absolutely irreplaceable -- but attention can be replaced with almost any other layer that shares information between tokens, such as pooling, convolution, random mixing, etc.

hm, residual is what I would not expect, can you elaborate why?

Re: Gemini Diffusion

#229

Post by [deleted] »

airstrike wrote: post #221
lukan wrote: post #219

Earlier quoted context omitted.

"At each stage, you get enough feedback to judge the right direction for the next iteration." Depends on the project I would say. What do you do, if all of a sudden the requirements change again? Or the plattform evolved/degraded? Then you compromise - and I can better compromise with simple solution. And I would never claim simple equals easy. Rather the opposite. Like you said, it is easy to make complex things. Al…

Re: this whole conversation, you might find this quick video a worthwhile watch https://www.youtube.com/watch?v=wrwxC9taL8w

[deleted]

Re: Gemini Diffusion

#230

Post by janalsncm »

moralestapia wrote: post #134
janalsncm wrote: post #94

Earlier quoted context omitted.

Let’s suppose we have 10k possible tokens in the vocabulary. Then text would be an image 10k pixels tall and N pixels wide, where N is the length of the text. For each column, exactly 1 pixel is white (corresponding to the word which is there) and the rest are black. Then the diffusion process is the same. Repeatedly denoising.

No, that intuition is incorrect. Denoising models work because a lot of regions turn out to be smooth, you cannot do that "in a discrete way" if that makes sense.

Feel free to give a better explanation. I am not an expert. Clearly denoising models do work on text though.
Post reply on HN