Diffusion Beats Autoregressive in Data-Constrained Settings
blog.ml.cmu.edu
Diffusion Beats Autoregressive in Data-Constrained Settings
1–10 of 16 posts
Re: Diffusion Beats Autoregressive in Data-Constrained Settings
#2[deleted]
Re: Diffusion Beats Autoregressive in Data-Constrained Settings
#3I have a feeling this technique might make waves: https://openreview.net/forum?id=c05qIG1Z2B#discussion
Re: Diffusion Beats Autoregressive in Data-Constrained Settings
#4I fail to understand why we would lack data. Sure, there is limited (historical) text, but if we just open up all available video, and send out interactive robots into the world, we'll drown in data. Then there is simulated data, and tons of sensors that can capture vast amounts of even more data.
Edit: from the source [1], this quote pretty much sums it all up: "Our 2022 paper predicted that high-quality text data would be fully used by 2024, whereas our new results indicate that might not happen until 2028."
[1] https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-...
Re: Diffusion Beats Autoregressive in Data-Constrained Settings
#5I fail to understand why we would lack data. Sure, there is limited (historical) text, but if we just open up all available video, and send out interactive robots into the world, we'll drown in data. Then there is simulated data, and tons of sensors that can capture vast amounts of even more data. Edit: from the source [1], this quote pretty much sums it all up: "Our 2022 paper predicted that high-quality text data w…
>send out interactive robots into the world
Easier said than done.
Robotics tends to be even more data-constrained than NLP. The real world only runs at 1x speed, and if your robot breaks something it costs real money. Simulators are simplistic compared to reality and take a lot of manual effort to build.
You will always need to make efficient use of the data you have.
Re: Diffusion Beats Autoregressive in Data-Constrained Settings
#6This paper was just too overhyped by the authors. Also, the initial evals were very limited and very strange. This blog post does a much better job at a similar observation -- goes into details and does proper evaluation (also better attribution): https://jinjieni.notion.site/Diffusion-Language-Models-are-S...
Re: Diffusion Beats Autoregressive in Data-Constrained Settings
#7I have a feeling this technique might make waves: https://openreview.net/forum?id=c05qIG1Z2B#discussion
There are definitely parallels between diffusion and reasoning models, mostly being able to spend longer to get a better solution by using a more precise ODE solver for diffusion or using more tokens for reasoning.
However, due to how diffusion models are trained, they never see their own predictions as input, so they cannot learn to store information across steps. This is the complete opposite for reasoning models.
Re: Diffusion Beats Autoregressive in Data-Constrained Settings
#8 > This paper addresses the challenge by asking: how can we trade off more compute for less data?
There is evidence that training RNN models that compute several steps with same input and coefficients (but different state) lead to better performance. It was shown in a followup to [1] that performed ablation study.
[1] https://arxiv.org/abs/1611.06188
They fixed number of time steps instead of varying it, and got better results.
Unfortunately, I forgot the title of that ablation paper.
Re: Diffusion Beats Autoregressive in Data-Constrained Settings
#9I have a feeling this technique might make waves: https://openreview.net/forum?id=c05qIG1Z2B#discussion
There are definitely parallels between diffusion and reasoning models, mostly being able to spend longer to get a better solution by using a more precise ODE solver for diffusion or using more tokens for reasoning. However, due to how diffusion models are trained, they never see their own predictions as input, so they cannot learn to store information across steps. This is the complete opposite for reasoning models.
You can train a diffusion model using its own predictions as input, no problem at all.
Re: Diffusion Beats Autoregressive in Data-Constrained Settings
#10> This paper addresses the challenge by asking: how can we trade off more compute for less data? Autoregressive models are not matched by compute and this is the major drawback. There is evidence that training RNN models that compute several steps with same input and coefficients (but different state) lead to better performance. It was shown in a followup to [1] that performed ablation study. [1] https://arxiv.org/ab…
Not sure if you meant this because it doesn't cite the paper you mention, but it's a similar work: "An Investigation of Model-Free Planning", Guez et Al. (Deepmind) 2019 https://arxiv.org/abs/1901.03559