Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

github.com

Post reply on HN
Print view View original

21–30 of 51 posts

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#21

Post by 0-_-0 »

ilaksh wrote: post #3
FranckDernoncou wrote: post #2

Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author. Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the b…

Amazing. Is it possible to do this with Qwen 3.6 27B? Will it work with quants (I assume so)?

3.6 already supports multi token generation AFAIK

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#22

Post by foobar10000 »

littlestymaar wrote: post #11
bertili wrote: post #8

Does this translate into a similar reduction in compute? What's the catch?

> Does this translate into a similar reduction in compute? No, quite the opposite actually. Like with speculative decoding this model will compute more tokens and discard the invalid ones. > What's the catch? LLMs[1] are limited by memory latency and not by compute[2]: because they process tokens one at a time, you spend more time loading and unloading the weights on the GPU registers from VRAM than waiting for compu…

Minor nit re[2]: for agentic workloads that are actually worth money - i.e., claude code and similar, things are either prefill-bound - which this does not help - or more importantly tps/user bound (at 150k+ context windows) - you want your big magic model to emit 200 tps/user. This is why Nvidia bought Groq (now LPU) and what Cerebras is trying to do, etc, etc. So for the stuff that makes money in the field - GPUs are not really compute bound once context lengths are large - but still memory transfer bound (may be KV-cache transfer, may be HBM->SRAM-on-chip, etc..)

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#23

Post by jbellis »

FranckDernoncou wrote: post #2

Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author. Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the b…

Really cool work!

Does the training data budget scale with model size?

How would you compare the Gemma 4 draft model which is also integrated with the base kv cache?

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#25

Post by kreelman »

dot_treo wrote: post #15
kreelman wrote: post #13

Earlier quoted context omitted.

Fantastic results. Well done. ...So this is built into the way the model works.. if I'm understanding it correctly. I was wondering what would be involved in getting it to work with GGUF files, rather than safetensor files...

Just to get it into a GGUF file would be fairly trivial. But using that GGUF file would need a bunch of additional things. One would need to create a new architecture derived from Qwen3, and then probably adapt the speculative decoding functionality. At the moment not even MTP is merged into llama.cpp, so I wouldn't quite hold my breath for it.

I thought that might be the case. I naively wondered. I'll see if I can understand the paper :-)

Hope the paper gets lots of references and the technique gets a lot of use to save power and time.

There's been several potential big changes for LLM inference efficiency over the last few months. There's been Attention Sequencing (I think it's called..?) Turbo Quant and this one.

Interesting times.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#26

Post by jbellis »

FranckDernoncou wrote: post #2

Paper: https://arxiv.org/abs/2605.12825 ; Code+models: https://github.com/chiennv2000/orthrus ; Disclosure: co-author. Idea: Inject a trainable diffusion attention module into each layer of a frozen AR Transformer. Both heads share one KV cache. Diffusion head projects K=32 tokens in parallel; AR head verifies in a second pass and accepts the longest matching prefix. Output distribution is provably identical to the b…

BTW the paper says

> Since only (Qdiff,Kdiff,Vdiff) are updated during training, the total number of trainable parameters is approximately 16% of the full model.

But the code defines q_proj_diff, k_proj_diff, v_proj_diff, and o_proj_diff, and it only matches 16% when you include the O term.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#27

Post by jbellis »

0-_-0 wrote: post #21
ilaksh wrote: post #3

Earlier quoted context omitted.

Amazing. Is it possible to do this with Qwen 3.6 27B? Will it work with quants (I assume so)?

3.6 already supports multi token generation AFAIK

Yes, but not diffusion based, it's still doing token-at-a-time speculation.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#29

Post by Xorlev »

foobar10000 wrote: post #22
littlestymaar wrote: post #11

Earlier quoted context omitted.

> Does this translate into a similar reduction in compute? No, quite the opposite actually. Like with speculative decoding this model will compute more tokens and discard the invalid ones. > What's the catch? LLMs[1] are limited by memory latency and not by compute[2]: because they process tokens one at a time, you spend more time loading and unloading the weights on the GPU registers from VRAM than waiting for compu…

Minor nit re[2]: for agentic workloads that are actually worth money - i.e., claude code and similar, things are either prefill-bound - which this does not help - or more importantly tps/user bound (at 150k+ context windows) - you want your big magic model to emit 200 tps/user. This is why Nvidia bought Groq (now LPU) and what Cerebras is trying to do, etc, etc. So for the stuff that makes money in the field - GPUs a…

> i.e., claude code and similar, things are either prefill-bound

When accounting for prefix caching, this greatly accelerates each turn. Barring large file reads, prefill still isn't the bottleneck vs. decoding reasoning tokens. Script-writing too.

This is especially true during exploration phases when traversing through directory trees and grepping files, you're talking about a few hundred tokens/turn.

Re: Orthrus-Qwen3: up to 7.8×tokens/forward on Qwen3, identical output distribution

#30

Post by hedgehog »

DeathArrow wrote: post #16

If someone can make this work with GGUF and Quantized Qwen 3.6 or Deepseek 4 it would greatly help running local models.

Multi-token prediction is available now, I'm still getting it set up but it sounds like it'll be 1.5x or 2x on the bigger models.
Post reply on HN