Isn't this basically the diffusion-autoregressive sampling strategy from the LLaDA paper, maybe more carefully evaluated?
the LLaDA paper is a scaled-up version of this paper; they cite it as an anonymous ICLR submission
Block Diffusion: Interpolating between autoregressive and diffusion models
21–30 of 37 posts
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#22Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#23Diffusion on images is easy to understand for me: you start with noise, the model denoises by shifting the pixels towards their final value. What is the equivalent operation for increasing or reducing noise in language here? Is the "noisy" sentence half-way through training or inference sort-of-correct but not really, and at 90% almost-correct but with slightly wrong words (semantically)? Is the noise somehow semanti…
For LLM, standard method to provide to foundation model sentence without one word (random, but already known to model) and ask to fill this word.
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#24this animation really makes the difference hit home: https://x.com/_akhaliq/status/1900027075370586262
This animation is a perfect abstract!
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#25Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#26Isn't this basically the diffusion-autoregressive sampling strategy from the LLaDA paper, maybe more carefully evaluated?
the LLaDA paper is a scaled-up version of this paper; they cite it as an anonymous ICLR submission
I'm not sure if this is what you mean, but LLaDA isn't block text diffusion. This is a mix between an autoregressive model and a diffusion model, which is brand new.
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#27Diffusion on images is easy to understand for me: you start with noise, the model denoises by shifting the pixels towards their final value. What is the equivalent operation for increasing or reducing noise in language here? Is the "noisy" sentence half-way through training or inference sort-of-correct but not really, and at 90% almost-correct but with slightly wrong words (semantically)? Is the noise somehow semanti…
For LLM, standard method to provide to foundation model sentence without one word (random, but already known to model) and ask to fill this word.
How is that diffusion? What noise is it denoising there?
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#28Earlier quoted context omitted.
For LLM, standard method to provide to foundation model sentence without one word (random, but already known to model) and ask to fill this word.
How is that diffusion? What noise is it denoising there?
Example, as I understand:
(I'm not sure how should look prompt, my guess): Prompt: answer, what word is missing in text query. Query: What is it denoising there?
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#29this animation really makes the difference hit home: https://x.com/_akhaliq/status/1900027075370586262
Imagine a whole git repo of a project materializing like that.
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#30Earlier quoted context omitted.
For LLM, standard method to provide to foundation model sentence without one word (random, but already known to model) and ask to fill this word.
How is that diffusion? What noise is it denoising there?
Don't think about defusion as denoising, but rather as learning a delta operator, or rather, the inverse of one. I don't know how diffusion language models work precisely, but if I were to haphazard a guess, I would say you may think of a sentence as a matrix of values, the operator as simply filling a line with zeros, and the inverse — what the model learns — as adding it back.
This is equivalent to cutting an image in blocks, and learning how to generate incrementally images by inpainting missing blocks. This in-painting mind you can be generated in multiple steps, where you incrementally add more into the block.