Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consistency diffusion language models: Up to 14x faster, no quality loss

together.ai

Post reply on HN
Print view View original

91–100 of 101 posts

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#91

Post by naasking »

abeppu wrote: post #74
naasking wrote: post #70

Earlier quoted context omitted.

IIRC, some researchers are working on mixed AR+diffusion models for this sort of thing.

I think the gap is, if they're building hybrids with _forward_ AR and diffusion, they risk giving up the cool part of diffusion which is reasoning back. I may be imposing unreasonable human biases on to this, but I really think it would be interesting to have the model engage with the structure of the text, rather than just being either a sequence or an array of tokens. E.g. "I'm going to _ tomorrow." If the _ is not…

I was thining of something like LLaDa that uses a Transformer to predict forward masked tokens:

https://arxiv.org/abs/2502.09992

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#92

Post by fumeux_fume »

jmalicki wrote: post #61
fumeux_fume wrote: post #60

Seeing half of an AR LLM's output tokens go to generating a predefined json schema bothers me so much. I would love to have an option to use diffusion for infilling.

One trick I learned for this was to use csv for LLM I/I and translate json csv at the boundary layer

Oh neat. So have the llm output csv instead of JSON and then convert it? How would handle nested structures?

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#93

Post by abeppu »

LarsDu88 wrote: post #78
abeppu wrote: post #62

Diffusion model papers are always interesting to read but I always feel like they need some mechanism to insert or delete tokens. In the example in the figure in this post, once it has fixed "British munchkin cats _ _ and ..." you _can't_ get to "British munchkin cats are a new and controversial breed." because there's not the right number of tokens between "cats" and "and". In a coding context, if your model samples…

This blogpost references block diffusion which fixes this issue that you are describing.

The cat example is from the section on their block-causal attention mask. I really don't think this fixes the issue. So far as I can see, the block schedule dictates when they sample at each position. It does _not_ change that they basically have an array-of-token-vars representation, and once `t_i` is sampled, nothing can "move" that value left or right.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#94

Post by jmalicki »

fumeux_fume wrote: post #92
jmalicki wrote: post #61

Earlier quoted context omitted.

One trick I learned for this was to use csv for LLM I/I and translate json csv at the boundary layer

Oh neat. So have the llm output csv instead of JSON and then convert it? How would handle nested structures?

Depending on how it's nested, you could denormalize, think of how you could denormalize a one-to-many SQL relationship

So if you have a user that has many automobiles, maybe instead of Autos: [...] you could do Auto1Make Auto2Make etc.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#95

Post by adgjlsfhk1 »

turnsout wrote: post #58
adgjlsfhk1 wrote: post #54

Earlier quoted context omitted.

opus 4.6 was going to be sonet 5 up until week of release. The price bump is even bigger than you realize because they don't let you run opus 4.6 at full speed unless you pay them an extra 10x for the new "fast mode"

If that's true, it would be surprising; the current Sonnet 4.6 is not in the same league as either Opus 4.5 or 4.6, either anecdotally or on benchmarks.

why is that surprising?

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#96

Post by turnsout »

adgjlsfhk1 wrote: post #95
turnsout wrote: post #58

Earlier quoted context omitted.

If that's true, it would be surprising; the current Sonnet 4.6 is not in the same league as either Opus 4.5 or 4.6, either anecdotally or on benchmarks.

why is that surprising?

Because Opus 4.6 is better than 4.5. So if it's true that Sonnet 5 was so good they gave it the Opus name, does that mean there was an Opus upgrade that didn't pan out? And what is Sonnet 4.6? An upgraded Haiku? Just trying to follow the red yarn in the conspiracy board here.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#97

Post by nl »

aurareturn wrote: post #13
nl wrote: post #12

Releasing this on the same day as Taalas's 16,000 token-per-second acceleration for the roughly comparable Llama 8B model must hurt! I wonder how far down they can scale a diffusion LM? I've been playing with in-browser models, and the speed is painful. https://taalas.com/products/

Nothing to do with each other. This is a general optimization. Taalas' is an ASIC that runs a tiny 8B model on SRAM. But I wonder how Taalas' product can scale. Making a custom chip for one single tiny model is different than running any model trillions in size for a billion users. Roughly, 53B transistors for every 8B params. For a 2T param model, you'd need 13 trillion transistor assuming scale is linear. One chip…

> Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM. It is expected in our labs this spring and will be integrated into our inference service shortly thereafter.

> Following this, a frontier LLM will be fabricated using our second-generation silicon platform (HC2). HC2 offers considerably higher density and even faster execution. Deployment is planned for winter.

From https://taalas.com/the-path-to-ubiquitous-ai/

Personally I think anything around the level of Sonnet 4.5 is worth burning to silicon because agentic workflows work. There are plenty of places where spending $50,000 for that makes sense (I have no idea of the pricing though)

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#98

Post by adgjlsfhk1 »

turnsout wrote: post #96
adgjlsfhk1 wrote: post #95

Earlier quoted context omitted.

why is that surprising?

Because Opus 4.6 is better than 4.5. So if it's true that Sonnet 5 was so good they gave it the Opus name, does that mean there was an Opus upgrade that didn't pan out? And what is Sonnet 4.6? An upgraded Haiku? Just trying to follow the red yarn in the conspiracy board here.

I don't know whether there was an opus that ran into trouble or if they just looked at the model they had and decided that they could charge more than originally intended. sonet 4.6 presumably is either a version of sonet 4.5 with optimizations for cost instead of perf (or a haiku that also got upscaled). Anthropic is preparing for IPO this year, so it's not exactly a stretch to suggest that they might be trying to decrease their losses and increase inference margin.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#99

Post by ak2600 »

aurareturn wrote: post #44
spuz wrote: post #30

Earlier quoted context omitted.

The 2.5kW figure is for a server running 10 HC1 chips: > The first generation HC1 chip is implemented in the 6 nanometer N6 process from TSMC. ... Each HC1 chip has 53 billion transistors on the package, most of it very likely for ROM and SRAM memory. The HC1 card burns about 200 watts, says Bajic, and a two-socket X86 server with ten HC1 cards in it runs 2,500 watts. https://www.nextplatform.com/2026/02/19/taalas-et…

I’m confused then. They need 10 of these to run an 8B model?

No. 250 watts to run an 8B model.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#100

Post by bjt12345 »

bob1029 wrote: post #19
bjt12345 wrote: post #15

I do wonder why diffusion models aren't used alongside constraint decoding for programming - surely it makes better sense then using an auto-regressive model.

Diffusion models need to infer the causality of language from within a symmetric architecture (information can flow forward or backward). AR forces information to flow in a single direction and is substantially easier to control as a result. The 2nd sentence in a paragraph of English text often cannot come before the first or the statement wouldn't make sense. Sometimes this is not an issue (and I think these are cas…

But I do wonder if diffusion models will be used in more complex Software Architecture for their long-term coherence, no exposure bias, and their symmetric architecture could work well with interaction nets.
Post reply on HN