This team is in a tough spot - they'll need to innovate and get their models in a good spot quickly. They are racing the clock against various similarly sized models being burned into ASICs and running at a similar speed (or faster).
Mercury 2.5
51–58 of 58 posts
Re: Mercury 2.5
#52The idea of getting a speed up in language models from using diffusion is compelling, but it just doesn’t seem like discrete diffusion models work as good at discrete non-diffusion. Which kind of makes sense, tokens don’t have implicit continuous representations that are easily noised. You have to learn a continuous embedding and figure out how to noise that, which is ideal for distillation (Ie the Gemma models, drafter heads) but not necessarily for pretraining.
There’s been more recent work on continuous space diffusion models for language this year. Sander Dielman has a good blog post on that.
But if I’d predict where diffusion lands in LLMs, it’ll be used in looped models like Astra. Once reasoning is happening in hidden states, we’re in a good continuous domain, perfect for diffusion. We’re going to end up swapping “looping” for predicting the models hidden states at the next “timestep” with diffusion. And that way the time to generate traces (no longer human intelligible though) will become 10x faster.
Re: Mercury 2.5
#53Ridiculously fast on OpenRouter, just subjectively it's a really strange experience because I've never seen a model respond or execute that quickly.
Try https://chatjimmy.ai/ from Taalas. There is an emergent space for super-fast-models esp finetuned or guardrailed to solve very specific latency sensitive tasks.
Generation is insanely fast, the other side is presentation, which can be slower and more controlled. Blasting the end-user with text walls is a UX problem now.
Re: Mercury 2.5
#54The benchmark comparison to other models was suspiciously missing. The self-comparison is a good representation of progress, but it is light years behind frontier models. Speed is great, but wrong is much worse than slow, IMO.
Re: Mercury 2.5
#55This team is in a tough spot - they'll need to innovate and get their models in a good spot quickly. They are racing the clock against various similarly sized models being burned into ASICs and running at a similar speed (or faster).
They can run it on „commodity” hardware as gpus, that gives them ability to change direction fast without burning money on ASIC? The space moves fast so model on ASIC can be outdated in couple of months?
Re: Mercury 2.5
#56Re: Mercury 2.5
#57The idea of getting a speed up in language models from using diffusion is compelling, but it just doesn’t seem like discrete diffusion models work as good at discrete non-diffusion. Which kind of makes sense, tokens don’t have implicit continuous representations that are easily noised. You have to learn a continuous embedding and figure out how to noise that, which is ideal for distillation (Ie the Gemma models, draf…
A lot of current diffusion LLMs don't convert tokens to continous space before noising them. They add discrete "noise", which is often as simple as replacing some tokens with [MASK].
The real problem is that when using fewer sampling steps than output tokens, diffusion formulations fundamentally cannot represent distributions where output tokens are heavily codependent. Autoregressive formulations don't have this problem, they can represent any distribution (ignoring limitations of the underlying model).
Re: Mercury 2.5
#58Earlier quoted context omitted.
Not just a model - It's a diffusion model. Instead of Next Token prediction it builds the entire page at once and then denoises it. Kind of mind blowing when you watch it happen.
Where can you watch it happen? Is there a video visualizing the diffusion process on text?
Just go type questions into the playground - leave "diffusion effect" on - you can see it for every question. https://chat.inceptionlabs.ai/