Block Diffusion: Interpolating between autoregressive and diffusion models
1–10 of 37 posts
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#2Only starts approaching PPL of AR at block size 4. May as well just use multi-token prediction with a standard AR model.
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#3The memory bandwidth bottleneck limits the speed of running local models, the fact that this model is parallelizable means that even with one batch inference it will be possible to balance memory bandwidth bottleneck and compute bottleneck (aka much more speed).
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#4I wonder how sliding window would work over blocks.
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#5This is cool but I feel like you lose the best part of language-diffusion models which is their ability to edit early tokens.
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#6Isn't this basically the diffusion-autoregressive sampling strategy from the LLaDA paper, maybe more carefully evaluated?
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#7this animation really makes the difference hit home: https://x.com/_akhaliq/status/1900027075370586262
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#8I love when someone comes up with a good idea that becomes immediately obvious as soon as it is introduced
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#9I wonder how sliding window would work over blocks.
Excellent question
Re: Block Diffusion: Interpolating between autoregressive and diffusion models
#10Based on the animation, I personally don't expect this to be very helpful. The main way diffusion models help is preventing answers like "No. [proceeds to explain why the answer is yes]", and since the blocks are so small, the LLM can't fully explain before it has to say yes or no.