Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consistency diffusion language models: Up to 14x faster, no quality loss

together.ai

Post reply on HN
Print view View original

71–80 of 101 posts

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#71

Post by moralestapia »

abeppu wrote: post #62

Diffusion model papers are always interesting to read but I always feel like they need some mechanism to insert or delete tokens. In the example in the figure in this post, once it has fixed "British munchkin cats _ _ and ..." you _can't_ get to "British munchkin cats are a new and controversial breed." because there's not the right number of tokens between "cats" and "and". In a coding context, if your model samples…

I think that having an early draft of the output is part of the appeal of this type of models.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#72

Post by abeppu »

stavros wrote: post #67
abeppu wrote: post #63

Earlier quoted context omitted.

I think maybe there are subsets of problems where you can have either a human or a smart LLM write a verifier (e.g. a property-based test?) and a performance measurement and let the dumb models generate candidates iterate on candidates?

Yeah, maybe, but then it would make much more sense to run a big model than hope one of the small ones randomly stumbles upon the solution, just because the possibility space is so much larger than the number of dumb LLMs you can run.

I don't work this way, so this is all a hypothetical to me, but the possibility space is larger than _any_ model can handle; models are effectively applying a really complex prior over a giant combinatorial space. I think the idea behind a swarm of small models (probably with higher temperature?) on a well-defined problem is akin to e.g. multi-chain MCMC.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#73

Post by abeppu »

moralestapia wrote: post #71
abeppu wrote: post #62

Diffusion model papers are always interesting to read but I always feel like they need some mechanism to insert or delete tokens. In the example in the figure in this post, once it has fixed "British munchkin cats _ _ and ..." you _can't_ get to "British munchkin cats are a new and controversial breed." because there's not the right number of tokens between "cats" and "and". In a coding context, if your model samples…

I think that having an early draft of the output is part of the appeal of this type of models.

Early draft yes. But when you write an early draft of prose or code, you leave yourself the ability to insert or remove material in a way that _changes the indexes of the tokens you already put in your draft_. If you write a letter, you may know that it ends with "Yours Truly, ", but not know the absolute number of tokens the letter will use. In this framework, once you say that "Yours Truly, John Hancock" are tokens 501 to 506, infilling the preceding sentences requires that you exactly preserve the number of tokens before that point ... which to me seems silly. I'm sure it's computationally messy to be able to slide stuff around, but if it meaningfully changes the topology of the search process, it may be worth it.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#74

Post by abeppu »

naasking wrote: post #70
abeppu wrote: post #62

Diffusion model papers are always interesting to read but I always feel like they need some mechanism to insert or delete tokens. In the example in the figure in this post, once it has fixed "British munchkin cats _ _ and ..." you _can't_ get to "British munchkin cats are a new and controversial breed." because there's not the right number of tokens between "cats" and "and". In a coding context, if your model samples…

IIRC, some researchers are working on mixed AR+diffusion models for this sort of thing.

I think the gap is, if they're building hybrids with _forward_ AR and diffusion, they risk giving up the cool part of diffusion which is reasoning back. I may be imposing unreasonable human biases on to this, but I really think it would be interesting to have the model engage with the structure of the text, rather than just being either a sequence or an array of tokens. E.g. "I'm going to _ tomorrow." If the _ is not just a token but an expansion in context, which might be a noun phrase, a verb phrase etc, it could be filled in with "the mall", "practice guitar". In code "if (_1) { return _2; }", _1 could be an expression whose type is bool, and which makes sense as a check to confirm that some process is finished. I don't care specifically how many tokens either of those is, but I do care that it makes sense in context.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#75

Post by sva_ »

irthomasthomas wrote: post #43
wongarsu wrote: post #28

Earlier quoted context omitted.

Notice how all the major AI companies (at least the ones that don't do open releases) stopped telling us how many parameters their models have. Parameter count was used as a measure for how great the proprietary models were until GPT3, then it suddenly stopped. And how inference prices have come down a lot, despite increasing pressure to make money. Opus 4.6 is $25/MTok, Opus 4.1 was $75/MTok, the same as Opus 4 and…

You're hitting on something really important that barely gets discussed. For instance, notice how opus 4.5's speed essentially doubled, bringing it right in line with the speed of sonnet 4.5? (sonnet 4.6 got a speed bump too, though closer to 25%). It was the very first thing I noticed: it looks suspiciously like they just rebranded sonnet as opus and raised the price. I don't know why more people aren't talking abou…

> a $20 floor and a $200 cap, no exceptions

Google caps at $250

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#76

Post by refulgentis »

impossiblefork wrote: post #20
blurbleblurble wrote: post #6

Earlier quoted context omitted.

Diffusion language models seem poised to smash purely autoregressive models. I'm giving it 1-2 years.

One appeal of it is for RL. If it ends up being a lot faster for generation, you'll be able to do a lot more RL. If people can make RL scalable-- make it so that RL isn't just a final phase, but something which is as big as the supervised stuff, then diffusion models are going to have an advantage. If not, I think autoregressive models will still be preferred. Diffusion models become fixed very fast, they can't actua…

> If not, I think autoregressive models will still be preferred. Diffusion models become fixed very fast, they can't actually refine their outputs, so we're not talking about some kind of refinement along the lines of: initial idea -> better idea -> something actually sound.

I'm really curious about this, I'm but a simple client developer, so I don't actually grok some of the differences.

For lack of a better word, there's a "normie" position that "omg diffusion means it can edit!!111! big unlock!" -- I think that's cute but I also don't see it as intuitively correct. And I guess I don't even know why I don't see it that way. But regardless, it sounds like I'm correct there.

> If not, I think autoregressive models will still be preferred.

But here I get lost, at least so far, diffusion models seem strictly significantly faster, and on par with models with the same parameter count.

If that is the case, why would autoregressive models still be preferred?

Asking this also makes me realize I am treating "diffusion models are better" as a premise, if I'm asserting they're always faster and ~same quality...

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#77

Post by crystal_revenge »

abeppu wrote: post #62

Diffusion model papers are always interesting to read but I always feel like they need some mechanism to insert or delete tokens. In the example in the figure in this post, once it has fixed "British munchkin cats _ _ and ..." you _can't_ get to "British munchkin cats are a new and controversial breed." because there's not the right number of tokens between "cats" and "and". In a coding context, if your model samples…

But the "infilling" problem isn't exactly solved for AR LLMs, so it's a strange critique.

Further more, you're applying the logic of AR LLMs to diffusion models. AR LLMs are only seeking the probability of the next token (a chain of conditional probability), but diffusion LLMs are modeling the probability of the entire output at once. Because of this token structures that leads to invalid outputs should be extremely low probability if properly trained.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#78

Post by LarsDu88 »

abeppu wrote: post #62

Diffusion model papers are always interesting to read but I always feel like they need some mechanism to insert or delete tokens. In the example in the figure in this post, once it has fixed "British munchkin cats _ _ and ..." you _can't_ get to "British munchkin cats are a new and controversial breed." because there's not the right number of tokens between "cats" and "and". In a coding context, if your model samples…

This blogpost references block diffusion which fixes this issue that you are describing.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#79

Post by robrenaud »

dakolli wrote: post #45
irthomasthomas wrote: post #43

Earlier quoted context omitted.

You're hitting on something really important that barely gets discussed. For instance, notice how opus 4.5's speed essentially doubled, bringing it right in line with the speed of sonnet 4.5? (sonnet 4.6 got a speed bump too, though closer to 25%). It was the very first thing I noticed: it looks suspiciously like they just rebranded sonnet as opus and raised the price. I don't know why more people aren't talking abou…

It kind of makes sense, at least a year or so ago, I know $20.00 unlimited plans were costing these companies ~$250.00 averaged out, they're still lighting money on fire with $200.00 but probably not nearly as bad, however, I'm not sure if costs have gone up with changes in models, seems like the agentic tooling is more expensive for them (hence why they're pushing anyone they can to pay per token).

Cite a source. Your concrete claim is that, on average, for every $1 of subscription revenue on a monthly subscription, OpenAI and Anthropic were losing $11.50?

It seems completely implausible.

I could believe that if a $20 sub used every possible token granted, it would cost $250. But certainly almost no one was completely milking their subscription. In the same way that no one is streaming netflix literally 24/7.

Re: Consistency diffusion language models: Up to 14x faster, no quality loss

#80

Post by irthomasthomas »

lumost wrote: post #69
irthomasthomas wrote: post #43

Earlier quoted context omitted.

You're hitting on something really important that barely gets discussed. For instance, notice how opus 4.5's speed essentially doubled, bringing it right in line with the speed of sonnet 4.5? (sonnet 4.6 got a speed bump too, though closer to 25%). It was the very first thing I noticed: it looks suspiciously like they just rebranded sonnet as opus and raised the price. I don't know why more people aren't talking abou…

It's quite plausible to me that the difference is inference configuration. This could be done through configurable depth, Moe experts, layers etc. Even beam decoding changes can make substantial performance changes. Train one large model, then down configure it for different pricing tiers.

I dont think thats plausible because they also just launched a high-speed variant which presumably has the inference optimization and smaller batching and costs about 10x

also, if you have inference optimizations why not apply them to all models?

Post reply on HN