61–70 of 101 posts
Seeing half of an AR LLM's output tokens go to generating a predefined json schema bothers me so much. I would love to have an option to use diffusion for infilling.
Earlier quoted context omitted.
Just tried this. Holy fuck. I'd take an army of high-school graduate LLMs to build my agentic applications over a couple of genius LLMs any day. This is a whole new paradigm of AI.
A billion stupid LLMs don't make a smart one, they just make one stupid LLM that's really fast at stupidity.
I wish there would be more of this research to speed things up rather than building ever larger models
Notice how all the major AI companies (at least the ones that don't do open releases) stopped telling us how many parameters their models have. Parameter count was used as a measure for how great the proprietary models were until GPT3, then it suddenly stopped. And how inference prices have come down a lot, despite increasing pressure to make money. Opus 4.6 is $25/MTok, Opus 4.1 was $75/MTok, the same as Opus 4 and…
For example, the Qwen3 technical report[1] says that the Qwen3 models are architecturally very similar to Qwen2.5, with the main change being a tweak in the attention layers to stabilize training. And if you compare table 1 in Qwen3 paper with table 1 in Qwen 2.5 technical report[2], the layer count, attention configuration and such is very similar. Yet Qwen3 was widely regarded as a significant upgrade to Qwen2.5.
However, for training, they doubled the pre-training token count, and tripled the number of languages. It's been shown that training on more languages can actually help LLMs generalize better. They used Qwen2.5 VL and Qwen 2.5 to generate additional training data by parsing a large number PDFs and turning them into high quality training tokens. They improved their annotation so they could more effectively provide diverse training tokens to the model, improving training efficiency.
They continued this trend with Qwen3.5, where even more and better training data[3] made their Qwen3.5-397B-A17B model match the 1T-parameter Qwen3-Max-Base.
That said there's also been a lot of work on model architecture[4], getting more speed and quality per parameter. In the case of Qwen3-Next architecture which 3.5 is based on, that means such things as hybrid attention for faster long-context operation, and sparse MoE and multi-token prediction for less compute per output token.
I used Qwen as an example here, from what I gather they're just an example of the general trend.
[1]: https://arxiv.org/abs/2505.09388
[2]: https://arxiv.org/abs/2412.15115
[3]: https://qwen.ai/blog?id=qwen3.5
[4]: https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d...
If this means there’s a 2x-7x speed up available to a scaled diffusion model like Inception Mercury, that’ll be a game changer. It feels 10x faster already…
Diffusion language models seem poised to smash purely autoregressive models. I'm giving it 1-2 years.
Earlier quoted context omitted.
Diffusion language models seem poised to smash purely autoregressive models. I'm giving it 1-2 years.
One appeal of it is for RL. If it ends up being a lot faster for generation, you'll be able to do a lot more RL. If people can make RL scalable-- make it so that RL isn't just a final phase, but something which is as big as the supervised stuff, then diffusion models are going to have an advantage. If not, I think autoregressive models will still be preferred. Diffusion models become fixed very fast, they can't actua…
Earlier quoted context omitted.
A billion stupid LLMs don't make a smart one, they just make one stupid LLM that's really fast at stupidity.
I think maybe there are subsets of problems where you can have either a human or a smart LLM write a verifier (e.g. a property-based test?) and a performance measurement and let the dumb models generate candidates iterate on candidates?
Earlier quoted context omitted.
Diffusion language models seem poised to smash purely autoregressive models. I'm giving it 1-2 years.
Didn't thinking tokens resolve the most problematic part of autoregressive models (the first few tokens set the constraints the model can't overcome later) and give it a massive advantage compared to diffusion models by showing the thinking trace? I can see diffusion models being used as a draft model to quickly predict a bunch of tokens and let the autoregressive model decide to use them or throw them away quickly,…
Check out this paper where they use diffusion during inference on the autoencoded prediction of an autoregressive model: https://openreview.net/forum?id=c05qIG1Z2B
Earlier quoted context omitted.
Notice how all the major AI companies (at least the ones that don't do open releases) stopped telling us how many parameters their models have. Parameter count was used as a measure for how great the proprietary models were until GPT3, then it suddenly stopped. And how inference prices have come down a lot, despite increasing pressure to make money. Opus 4.6 is $25/MTok, Opus 4.1 was $75/MTok, the same as Opus 4 and…
You're hitting on something really important that barely gets discussed. For instance, notice how opus 4.5's speed essentially doubled, bringing it right in line with the speed of sonnet 4.5? (sonnet 4.6 got a speed bump too, though closer to 25%). It was the very first thing I noticed: it looks suspiciously like they just rebranded sonnet as opus and raised the price. I don't know why more people aren't talking abou…
Train one large model, then down configure it for different pricing tiers.
Diffusion model papers are always interesting to read but I always feel like they need some mechanism to insert or delete tokens. In the example in the figure in this post, once it has fixed "British munchkin cats _ _ and ..." you _can't_ get to "British munchkin cats are a new and controversial breed." because there's not the right number of tokens between "cats" and "and". In a coding context, if your model samples…