Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consistency diffusion language models: Up to 14x faster, no quality loss

together.ai

Post reply on HN
Print view View original

31–40 of 101 posts

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#31

Post by LoganDark »

janalsncm wrote: post #23
LoganDark wrote: post #18

Can't wait for the day I can actually try a diffusion model on my own machine (128GB M4 Max) rather than as a hosted service. So far I haven't seen a single piece of software that supports it.

You can try it today. You can get them from huggingface. Here is an example: https://huggingface.co/tencent/WeDLM-8B-Instruct Diffusion isn’t natively supported in the transformers library yet so you have to use their custom inference code.

I don't subscribe to the Python craze, but this could be interesting. Thanks!

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#32

Post by stavros »

5o1ecist wrote: post #29
wongarsu wrote: post #28

Earlier quoted context omitted.

Notice how all the major AI companies (at least the ones that don't do open releases) stopped telling us how many parameters their models have. Parameter count was used as a measure for how great the proprietary models were until GPT3, then it suddenly stopped. And how inference prices have come down a lot, despite increasing pressure to make money. Opus 4.6 is $25/MTok, Opus 4.1 was $75/MTok, the same as Opus 4 and…

[flagged]

Why don't we need them? If I need to run a hundred small models to get a given level of quality, what's the difference to me between that and running one large model?

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#33

Post by stavros »

janalsncm wrote: post #22
yjftsjthsd-h wrote: post #4

Is anyone doing any form of diffusion language models that are actually practical to run today on the actual machine under my desk? There's loads of more "traditional" .gguf options (well, quants) that are practical even on shockingly weak hardware, and I've been seeing things that give me hope that diffusion is the next step forward, but so far it's all been early research prototypes.

I worked on it for a more specialized task (query rewriting). It’s blazing fast. A lot of inference code is set up for autoregressive decoding now. Diffusion is less mature. Not sure if Ollama or llama cpp support it.

How was the quality?

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#34

Post by somewhereoutth »

5o1ecist wrote: post #29
wongarsu wrote: post #28

Earlier quoted context omitted.

Notice how all the major AI companies (at least the ones that don't do open releases) stopped telling us how many parameters their models have. Parameter count was used as a measure for how great the proprietary models were until GPT3, then it suddenly stopped. And how inference prices have come down a lot, despite increasing pressure to make money. Opus 4.6 is $25/MTok, Opus 4.1 was $75/MTok, the same as Opus 4 and…

[flagged]

I'd suggest that a measure like 'density[1]/parameter' as you put it will asymptotically rise to a hard theoretical limit (that probably isn't much higher than what we have already). So quite unlike Moore's Law.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#35

Post by stavros »

LASR wrote: post #14
nl wrote: post #12

Releasing this on the same day as Taalas's 16,000 token-per-second acceleration for the roughly comparable Llama 8B model must hurt! I wonder how far down they can scale a diffusion LM? I've been playing with in-browser models, and the speed is painful. https://taalas.com/products/

Just tried this. Holy fuck. I'd take an army of high-school graduate LLMs to build my agentic applications over a couple of genius LLMs any day. This is a whole new paradigm of AI.

A billion stupid LLMs don't make a smart one, they just make one stupid LLM that's really fast at stupidity.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#36

Post by block_dagger »

5o1ecist wrote: post #29
wongarsu wrote: post #28

Earlier quoted context omitted.

Notice how all the major AI companies (at least the ones that don't do open releases) stopped telling us how many parameters their models have. Parameter count was used as a measure for how great the proprietary models were until GPT3, then it suddenly stopped. And how inference prices have come down a lot, despite increasing pressure to make money. Opus 4.6 is $25/MTok, Opus 4.1 was $75/MTok, the same as Opus 4 and…

[flagged]

Doesn’t the widely accepted Bitter Lesson say the exact opposite about specialized models vs generalized?

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#37

Post by spockz »

stavros wrote: post #32
5o1ecist wrote: post #29

Earlier quoted context omitted.

[flagged]

Why don't we need them? If I need to run a hundred small models to get a given level of quality, what's the difference to me between that and running one large model?

You can run smaller models on smaller compute hardware and split the compute. For large models you need to be able to fit the whole model in memory to get any decent throughput.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#38

Post by precompute »

nl wrote: post #12

Releasing this on the same day as Taalas's 16,000 token-per-second acceleration for the roughly comparable Llama 8B model must hurt! I wonder how far down they can scale a diffusion LM? I've been playing with in-browser models, and the speed is painful. https://taalas.com/products/

This is crazy!

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#39

Post by stavros »

spockz wrote: post #37
stavros wrote: post #32

Earlier quoted context omitted.

Why don't we need them? If I need to run a hundred small models to get a given level of quality, what's the difference to me between that and running one large model?

You can run smaller models on smaller compute hardware and split the compute. For large models you need to be able to fit the whole model in memory to get any decent throughput.

Ah interesting, I didn't realize MoE doesn't need to all run in the same place.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#40

Post by small_model »

nl wrote: post #12

Releasing this on the same day as Taalas's 16,000 token-per-second acceleration for the roughly comparable Llama 8B model must hurt! I wonder how far down they can scale a diffusion LM? I've been playing with in-browser models, and the speed is painful. https://taalas.com/products/

This is exceptionally fast (almost instant) whats the catch? Answer was there before I lifted return key!
Post reply on HN