Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consistency diffusion language models: Up to 14x faster, no quality loss

together.ai

Post reply on HN
Print view View original

21–30 of 101 posts

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#21

Post by tokenless »

LASR wrote: post #14
nl wrote: post #12

Releasing this on the same day as Taalas's 16,000 token-per-second acceleration for the roughly comparable Llama 8B model must hurt! I wonder how far down they can scale a diffusion LM? I've been playing with in-browser models, and the speed is painful. https://taalas.com/products/

Just tried this. Holy fuck. I'd take an army of high-school graduate LLMs to build my agentic applications over a couple of genius LLMs any day. This is a whole new paradigm of AI.

When that genrates 10k of output slop in less latency than my web server doing some crud shit....amazing!

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#22

Post by janalsncm »

yjftsjthsd-h wrote: post #4

Is anyone doing any form of diffusion language models that are actually practical to run today on the actual machine under my desk? There's loads of more "traditional" .gguf options (well, quants) that are practical even on shockingly weak hardware, and I've been seeing things that give me hope that diffusion is the next step forward, but so far it's all been early research prototypes.

I worked on it for a more specialized task (query rewriting). It’s blazing fast.

A lot of inference code is set up for autoregressive decoding now. Diffusion is less mature. Not sure if Ollama or llama cpp support it.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#23

Post by janalsncm »

LoganDark wrote: post #18

Can't wait for the day I can actually try a diffusion model on my own machine (128GB M4 Max) rather than as a hosted service. So far I haven't seen a single piece of software that supports it.

You can try it today. You can get them from huggingface. Here is an example:

https://huggingface.co/tencent/WeDLM-8B-Instruct

Diffusion isn’t natively supported in the transformers library yet so you have to use their custom inference code.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#26

Post by LarsDu88 »

meatmanek wrote: post #8
blurbleblurble wrote: post #6

Earlier quoted context omitted.

Diffusion language models seem poised to smash purely autoregressive models. I'm giving it 1-2 years.

Feels like the sodium ion battery vs lithium ion battery thing, where there are theoretical benefits of one but the other has such a head start on commercialization that it'll take a long time to catch up.

Not really. Unlike with physical goods like batteries, the hardware for training a diffusion vs an autoregressive language model is more or less exactly the same.

Although the lab that did this research (Chris Re and Tri Dao are involved) is run by the world's experts in squeezing CUDA and Nvidia hardware for every last drop of performance.

At the API level, the primary differences will be the addition of text infill capabilities for language generation. I also somewhat expect certain types of generation to be more cohesive (e.g. comedy or stories where you need to think of the punchline or ending first!)

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#28

Post by wongarsu »

MASNeo wrote: post #9

I wish there would be more of this research to speed things up rather than building ever larger models

Notice how all the major AI companies (at least the ones that don't do open releases) stopped telling us how many parameters their models have. Parameter count was used as a measure for how great the proprietary models were until GPT3, then it suddenly stopped.

And how inference prices have come down a lot, despite increasing pressure to make money. Opus 4.6 is $25/MTok, Opus 4.1 was $75/MTok, the same as Opus 4 and Opus 3. OpenAI's o1 was $60/MTok, o1 pro $600/MTok, gpt-5.2 is $14/MTok and 5.2-pro is $168/MTok.

Also note how GPT-4 was rumored to be in the 1.8T realm, and now Chinese models in the 1T realm can match or surpass it. And I doubt the Chinese have a monopoly on those efficiency improvements

I doubt frontier models have actually substantially grown in size in the last 1.5 years, and potentially have a lot fewer parameters than the frontier models of old

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#29

Post by 5o1ecist »

wongarsu wrote: post #28
MASNeo wrote: post #9

I wish there would be more of this research to speed things up rather than building ever larger models

Notice how all the major AI companies (at least the ones that don't do open releases) stopped telling us how many parameters their models have. Parameter count was used as a measure for how great the proprietary models were until GPT3, then it suddenly stopped. And how inference prices have come down a lot, despite increasing pressure to make money. Opus 4.6 is $25/MTok, Opus 4.1 was $75/MTok, the same as Opus 4 and…

[flagged]

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#30

Post by spuz »

aurareturn wrote: post #13
nl wrote: post #12

Releasing this on the same day as Taalas's 16,000 token-per-second acceleration for the roughly comparable Llama 8B model must hurt! I wonder how far down they can scale a diffusion LM? I've been playing with in-browser models, and the speed is painful. https://taalas.com/products/

Nothing to do with each other. This is a general optimization. Taalas' is an ASIC that runs a tiny 8B model on SRAM. But I wonder how Taalas' product can scale. Making a custom chip for one single tiny model is different than running any model trillions in size for a billion users. Roughly, 53B transistors for every 8B params. For a 2T param model, you'd need 13 trillion transistor assuming scale is linear. One chip…

The 2.5kW figure is for a server running 10 HC1 chips:

> The first generation HC1 chip is implemented in the 6 nanometer N6 process from TSMC. ... Each HC1 chip has 53 billion transistors on the package, most of it very likely for ROM and SRAM memory. The HC1 card burns about 200 watts, says Bajic, and a two-socket X86 server with ten HC1 cards in it runs 2,500 watts.

https://www.nextplatform.com/2026/02/19/taalas-etches-ai-mod...

Post reply on HN