Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
1–10 of 51 posts
Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
#2Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author.
Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the base model.
Results:
- Up to 7.8x TPF, ~6x wall-clock on MATH-500.
- 16% of params trained, - vs. diffusion LMs (Dream, Fast-dLLM-v2, SDAR, Mercury, Gemini Diffusion): they modify base weights and lose accuracy (Fast-dLLM-v2: -11 pts on MATH-500). Orthrus freezes the backbone; accuracy matches Qwen3-8B exactly.
- vs. Speculative Decoding (EAGLE-3, DFlash): no external drafter, no separate cache, zero TTFT penalty (no drafter to init/sync). KV overhead is O(1) (~4.5 MiB flat). Acceptance length on MATH-500: 11.7 vs. 7.9 (DFlash) vs. 3.5 (EAGLE-3).
- Single-step denoising beats multi-step (6.35 vs. 3.53 TPF). KL distillation beats CE on acceptance rate.
Limitations: strictly bounded by the frozen base model (inherits its biases, hallucinations, knowledge gaps); Qwen3-only evaluation; greedy + rejection sampling only.
Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
#3Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author. Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the b…
Amazing. Is it possible to do this with Qwen 3.6 27B? Will it work with quants (I assume so)?
Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
#4The most interesting part of this idea for me is how it wasn't tried / implemented before, as it makes sense.
I haven't read the paper but of course DTree tricks work here as well
Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
#5Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author. Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the b…
Amazing. Is it possible to do this with Qwen 3.6 27B? Will it work with quants (I assume so)?
From a quick and shallow view of the paper, it looks very feasible (with a little tinkering ) to be adapted to qwen3.6 27B. The process looks somewhat similar to training a LoRA, or in a way distilling your own model so that a mini model learns how to imitate it, and you glue them. I might bite the bullet and rent a gpu to do it for 3.6 27b, as this will solve a lot of my problems.
Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
#6Earlier quoted context omitted.
Amazing. Is it possible to do this with Qwen 3.6 27B? Will it work with quants (I assume so)?
From a quick and shallow view of the paper, it looks very feasible (with a little tinkering ) to be adapted to qwen3.6 27B. The process looks somewhat similar to training a LoRA, or in a way distilling your own model so that a mini model learns how to imitate it, and you glue them. I might bite the bullet and rent a gpu to do it for 3.6 27b, as this will solve a lot of my problems.
Scratch that, I don't have that kind of money, and 3.5's architecture is a little more divergent from 3's, so it will be a bit less trivial. It does look possible, just not on a student's paycheck.
Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
#7Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author. Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the b…
So, it's D-Flash but at each transformer layer and share the KV cache of the original model? Very smart!
Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
#8Does this translate into a similar reduction in compute?
What's the catch?
Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
#9Earlier quoted context omitted.
From a quick and shallow view of the paper, it looks very feasible (with a little tinkering ) to be adapted to qwen3.6 27B. The process looks somewhat similar to training a LoRA, or in a way distilling your own model so that a mini model learns how to imitate it, and you glue them. I might bite the bullet and rent a gpu to do it for 3.6 27b, as this will solve a lot of my problems.
Scratch that, I don't have that kind of money, and 3.5's architecture is a little more divergent from 3's, so it will be a bit less trivial. It does look possible, just not on a student's paycheck.
There are websites that let you rent GPUs for cheap, such as QuickPod. Have you checked those P2P GPU rentals out?
Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution
#10Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author. Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the b…
Do you plan on releasing the training code?