this animation really makes the difference hit home: https://x.com/_akhaliq/status/1900027075370586262
I wish every paper had a succinct image like that!
Block Diffusion: Interpolating between autoregressive and diffusion models
31–37 of 37 posts
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#32Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#33this animation really makes the difference hit home: https://x.com/_akhaliq/status/1900027075370586262
Imagine a whole git repo of a project materializing like that.
I'm really curious to see if this kind of diffusion works with "guided generation" / grammars as well. There's a lot of "structure" in code that you could use for minimising the low hanging fruit errors, and focus on the logic of the code instead.
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#34Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#35This is cool but I feel like you lose the best part of language-diffusion models which is their ability to edit early tokens.
Those early tokens aren't necessarily immutable, they still could be "edited" depending on UI. Human conversation and even internal compositional cogitation is full of "what I meant by that" or "on second thought" type clarifications and corrections. Sometimes these aren't verbosely disclaimed, there's body language involved. Likewise there could be occasional lookback parsing and later blocks could convey modificati…
[deleted]
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#36Earlier quoted context omitted.
the LLaDA paper is a scaled-up version of this paper; they cite it as an anonymous ICLR submission
I'm not sure if this is what you mean, but LLaDA isn't block text diffusion. This is a mix between an autoregressive model and a diffusion model, which is brand new.
It is a soft-block text diffusion. They have one super-block of fixed size loaded and then allow the model to only unmask tokens by going through the soft-blocks. As the source code is available and I was able to change it into an actual block diffusion, but as the model was trained only on super-blocks, it was always trying to generate eos tokens at each block end before I extended it. I've tried a few workarounds that half worked, but I guess a very small scale finetune is needed to resolve it fully.
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#37Only starts approaching PPL of AR at block size 4. May as well just use multi-token prediction with a standard AR model.
[deleted]