TiDAR: Think in Diffusion, Talk in Autoregression
1–10 of 24 posts
Re: TiDAR: Think in Diffusion, Talk in Autoregression
#2I've tried dLLMs like Mercury and they look promising.
Re: TiDAR: Think in Diffusion, Talk in Autoregression
#3An update to Gemini diffusion is one of my most eagerly anticipated AI releases. It released to mild fanfare (mostly because you needed to request access to use it), and there has been silence ever since.
Hopefully it's not more Google abandonware, because it was wicked fast and a delight to use
Re: TiDAR: Think in Diffusion, Talk in Autoregression
#4An update to Gemini diffusion is one of my most eagerly anticipated AI releases. It released to mild fanfare (mostly because you needed to request access to use it), and there has been silence ever since. Hopefully it's not more Google abandonware, because it was wicked fast and a delight to use
It's not a very promising direction because autoregressive LLMs still deliver better output quality per model weight, as a rule.
Now, is it possible that a model can combine advantages of both? Combine fast generation and multidirectional causality of diffusion with precision, capabilities and generalization of autoregression?
Maybe. This paper is research in that direction. So far, it's not a clear upgrade over autoregressive LLMs.
Re: TiDAR: Think in Diffusion, Talk in Autoregression
#5Diffusion is favored by current GPUs .
Over time we seem to have a tendency to build models that are well matched to our machines
Re: TiDAR: Think in Diffusion, Talk in Autoregression
#6An update to Gemini diffusion is one of my most eagerly anticipated AI releases. It released to mild fanfare (mostly because you needed to request access to use it), and there has been silence ever since. Hopefully it's not more Google abandonware, because it was wicked fast and a delight to use
It's not a very promising direction because autoregressive LLMs still deliver better output quality per model weight, as a rule. Now, is it possible that a model can combine advantages of both? Combine fast generation and multidirectional causality of diffusion with precision, capabilities and generalization of autoregression? Maybe. This paper is research in that direction. So far, it's not a clear upgrade over auto…
Diffusion LMs do seem to be able to get more out of the same data. In a world where we are already training transformer based LLMs on all text available, diffusion LMs ability to continue learning on a fixed set of data may be able to outperform transformers
Re: TiDAR: Think in Diffusion, Talk in Autoregression
#7An update to Gemini diffusion is one of my most eagerly anticipated AI releases. It released to mild fanfare (mostly because you needed to request access to use it), and there has been silence ever since. Hopefully it's not more Google abandonware, because it was wicked fast and a delight to use
It's not a very promising direction because autoregressive LLMs still deliver better output quality per model weight, as a rule. Now, is it possible that a model can combine advantages of both? Combine fast generation and multidirectional causality of diffusion with precision, capabilities and generalization of autoregression? Maybe. This paper is research in that direction. So far, it's not a clear upgrade over auto…
4-5 times faster with minimal change in quality seems like a clear upgrade in efficiency.
Re: TiDAR: Think in Diffusion, Talk in Autoregression
#8Earlier quoted context omitted.
It's not a very promising direction because autoregressive LLMs still deliver better output quality per model weight, as a rule. Now, is it possible that a model can combine advantages of both? Combine fast generation and multidirectional causality of diffusion with precision, capabilities and generalization of autoregression? Maybe. This paper is research in that direction. So far, it's not a clear upgrade over auto…
4-5 times faster with minimal change in quality seems like a clear upgrade in efficiency.
Latency may be better, but throughput (the thing companies care about) may be the same or worse, since every step the entire diffusion window has to be passed through the model. With AR models only the most recent token goes through, which is much more compute efficient allowing you to be memory bound. Trade off with these models is more than one token per forward pass, but idk the point where that becomes worth it (probably depends on model and diffusion window size)
Re: TiDAR: Think in Diffusion, Talk in Autoregression
#9Earlier quoted context omitted.
It's not a very promising direction because autoregressive LLMs still deliver better output quality per model weight, as a rule. Now, is it possible that a model can combine advantages of both? Combine fast generation and multidirectional causality of diffusion with precision, capabilities and generalization of autoregression? Maybe. This paper is research in that direction. So far, it's not a clear upgrade over auto…
Diffusion LMs do seem to be able to get more out of the same data. In a world where we are already training transformer based LLMs on all text available, diffusion LMs ability to continue learning on a fixed set of data may be able to outperform transformers https://arxiv.org/abs/2511.03276
There’s another paper that shows you can get the same effect by training auto regression on Fill in the middle data.
So it’s more about the mask modeling objective than Diffusion.
Re: TiDAR: Think in Diffusion, Talk in Autoregression
#10An update to Gemini diffusion is one of my most eagerly anticipated AI releases. It released to mild fanfare (mostly because you needed to request access to use it), and there has been silence ever since. Hopefully it's not more Google abandonware, because it was wicked fast and a delight to use
It's not a very promising direction because autoregressive LLMs still deliver better output quality per model weight, as a rule. Now, is it possible that a model can combine advantages of both? Combine fast generation and multidirectional causality of diffusion with precision, capabilities and generalization of autoregression? Maybe. This paper is research in that direction. So far, it's not a clear upgrade over auto…
> still deliver better output quality per model weight, as a rule.
is it possible to quantify that and just have a linked slider for quality and speed? If I can get an answer that's 80% right in 1/10th the time, and then iterate on that who comes out ahead?